Storage Systems and Methods for Robotic Picking

ABSTRACT

A mobile manipulator robot for retrieving inventory items from a storage system. The robot includes a body, a wheel assembly, a sensor to locate a position of the robot within the storage system, an interface configured to send processor readable data to a remote processor and to an operator interface, and receive processor executable instructions from the remote processor and from the operator interface, an imaging device to capture images of the inventory items, a picking manipulator, first and second pneumatic gripping elements for grasping the inventory items, and a coupler configured to mate with a valve to access a pneumatic supply for operating at least one of the first or second pneumatic gripping elements. The robot is configured to transition the valve from a closed condition to an open condition and selectively place one of the first or the second pneumatic gripping elements in communication with the pneumatic supply.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the filing date of U.S.Provisional Patent Application No. 62/961,390, filed Jan. 15, 2020 andthe benefit of the filing date of U.S. Provisional Patent ApplicationNo. 62/879,843, filed Jul. 29, 2019, the disclosures of which are eachhereby incorporated herein by reference.

BACKGROUND OF THE DISCLOSURE

The present disclosure relates generally to storage systems andinventory retrieval methods, and more particularly, to a storage systemand a mobile, manipulator robot for retrieving inventory items from thestorage system.

Warehouses, or distribution fulfillment centers, require systems thatenable the efficient storage and retrieval of a large number of diverseproducts. Traditionally, inventory items are stored in containers andarranged on rows of shelving on either side of an aisle. Each container,or bin, holds a plurality of items of one or more product types. Theaisles provide access between the shelving for an operator or robot tomigrate the aisles and retrieve the items. It is well understood thatthe aisles reduce the storage density of the system. In other words, theamount of space actually used for the storage of products (e.g., theshelving) is relatively small compared to the amount of space requiredfor the storage system as a whole. As warehouse space is often scarceand expensive, alternative storage systems that maximize storage spaceare desired.

In one alternative approach, which offers a significant improvement instorage density, containers are stacked on top of one another andarranged in adjacent rows. That is, no aisle is provided between theadjacent rows of stacked containers. Thus, more containers, and in turninventory, can be stored in a given space.

Various methods for retrieving inventory from the stacked containershave been contemplated. U.S. Pat. No. 10,189,641, for example, disclosesa system in which containers are stacked and arranged in a plurality ofrows underneath a grid. Vehicles equipped with a lifting apparatusnavigate the grid and lift a desired container. The container is thentransported down a port to a picking/sorting zone, where an operator orrobot picks individual products from the container and sorts theproducts into one or more order containers. To minimize unnecessarytransportation of the containers, each container is typicallytransported to the picking/sorting zone only after multiple orders of aspecific product have been received.

Despite the increased storage density provided by the known stackedstorage system, various shortcoming remain. For example, orderfulfilment times are often lengthy, particularly for products that areordered infrequently because the containers are retrieved in priority asa function of the number of products of one type that have been ordered.Additionally, the vehicles are required to navigate long distances(which takes considerable time and consumes considerable battery power)while driving bins back-and-fourth to the transportation ports.Furthermore, the required picking/sorting zones reduce the overallstorage density of the warehouse and add additional complexity andcosts. While the throughput of the stacked storage system can beincreased by adding additional vehicles to the grid (or by modifying thesystem to include additional container transportation ports), there is alimit to the amount of vehicles that can be operated on the grid beforethe grid becomes overly congested with vehicles and the throughput ofthe system declines due to gridlock.

BRIEF SUMMARY OF THE DISCLOSURE

In accordance with a first aspect of the present disclosure, a highdensity storage structure is provided. The storage structure includessupport members configured to house a plurality of containers, a firstset of parallel rails to support a mobile, manipulator robot and a fluidsupply line having a plurality of valves disposed within the supplyline. Each of the valves have a closed condition in which the supplyline is in fluid isolation from an outside environment and an opencondition in which the supply line is in fluid communication with theoutside environment such that a mobile, manipulator robot traversing thefirst set of parallel rails may receive a fluid supply from the fluidsupply line.

In accordance with another aspect of the disclosure, a mobile,manipulator robot for retrieving inventory from the storage structure isprovided. The robot may include a body having an interface configured tosend processor readable data to a central processor and receiveprocessor executable instructions from the central processor, a mobilityassembly coupled to the body, a coupler selectively mateable to a portto receive a fluid supply from a supply line, and a picking armconnected to the body. The picking arm may be coupled to a firstpneumatic gripping tool configured to grasp inventory items.

In accordance with yet another aspect of the disclosure, a method forcontrolling a mobile, manipulator robot to retrieve a product from acontainer located in a storage structure is provided. The method mayinclude moving the mobile, manipulator robot over a first set ofparallel rails of the storage structure and to a picking location,identifying a grasping region located on a product based at least inpart upon image data obtained by a sensor attached to the mobile,manipulator robot, adjusting a picking arm equipped with a pneumaticgripping tool to a grasping pose, and grasping the product using thepneumatic gripping tool.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic perspective view of a frame structure for housinga plurality of stacked containers according to the prior art.

FIG. 2 is a schematic plan view of a portion of the frame structure ofFIG. 1.

FIGS. 3A and 3B are schematic perspective views, from the rear and frontrespectively, of a load handling device according to the prior art foruse with the frame structure depicted in FIGS. 1 and 2.

FIG. 3C is a schematic perspective view depicting a container beinglifted by the load handling device of FIGS. 3A and 3B.

FIG. 4 is a schematic perspective view of the frame structure of FIG. 1having a plurality of the load handling devices of FIGS. 3A-3C installedon the frame structure.

FIG. 5 is a schematic perspective view of the frame structure of FIG. 4illustrating a digging operation to retrieve a target container from astack of containers.

FIG. 6A is a schematic illustration of a robotic system including astorage structure for housing a plurality of stacked containersaccording to an embodiment of the present disclosure.

FIG. 6B is a schematic perspective view of the storage structure of FIG.6A.

FIG. 6C is a schematic perspective view of two storage structuresarranged on top of one another according to an embodiment of the presentdisclosure.

FIG. 6D is a schematic side elevation view of a digging robot performinga digging operation within the storage structure of FIG. 6B.

FIG. 7A is a perspective view of a rail illustrating a channel extendingthrough the rail and a conduit extending from the channel to a surfaceof the rail.

FIG. 7B is an enlarged view of a portion of the rail of FIG. 7A.

FIG. 8A is a cross-section view of a valve located within the conduit ofFIG. 7A.

FIG. 8B is an enlarged view of the valve of FIG. 8A.

FIG. 9A is a schematic perspective view of a mobile, manipulator robotincluding a picking arm equipped with a pneumatic gripping tool and atool holder, installed on top of the storage structure of FIG. 6B.

FIG. 9B is an enlarged view of a portion of the mobile, manipulatorrobot of FIG. 9A.

FIG. 9C is a flow chart showing an example method of determining agrasping pose of the picking arm of the mobile, manipulator robot ofFIG. 9A.

FIG. 9D is a schematic illustration of a plurality of product itemslocated within a container.

FIG. 9E is a schematic illustration showing grasping regions of theproduct items of FIG. 9D.

FIG. 9F is a perspective view of a first set of pneumatic gripping toolsstored in the tool holder of FIG. 9A.

FIG. 9G is a perspective view of a second set of pneumatic grippingtools stored in the tool holder of FIG. 9A.

FIG. 10A is a top elevation view illustrating a mobility assembly of therobot of FIG. 9A.

FIG. 10B is a schematic view of a prop mechanism that assists inrotating the wheels of the mobility assembly of FIG. 10A.

FIG. 11 is a schematic cross-section view of a coupler of the robot ofFIG. 9A.

FIG. 12A is a perspective view of the picking arm of the robot of FIG.9A.

FIG. 12B is a side view of a portion of the picking arm of FIG. 12A.

FIG. 12C is a schematic cross-section view of an order bin and a targetcontainer holding inventory items of different sizes.

FIG. 13A is a cross-section view illustrating the coupling between thepneumatic gripping tool of FIG. 9A and the picking arm of FIGS. 12A and12B.

FIG. 13B is a schematic perspective view illustrating the couplingbetween the picking arm of FIGS. 12A and 12B and alternative pneumaticgripping tools.

FIG. 13C is a schematic illustration showing two pneumatic supply linesof the mobile, manipulator robot of FIG. 9A coupleable to severalexample pneumatic tools.

FIGS. 14A and 14B are schematic cross-sections illustrating the couplingbetween the coupler of FIG. 11 and the conduit of FIG. 7A.

FIG. 15 is a flow chart showing a method of grasping a product itemusing the picking arm and the pneumatic gripping tool of FIG. 13A.

FIG. 16A is a schematic perspective view of a mobile, manipulator robotincluding a container retrieval device having a hoist plate according toanother embodiment of the present disclosure.

FIG. 16B is a schematic perspective view of the hoist plate of FIG. 16A.

FIG. 16C is a schematic perspective view of a hoist plate including aplurality of suction cups according to another embodiment of the presentdisclosure.

FIGS. 16D and 16E are schematic perspective views of a hoist plateincluding a retractable and moveable picking arm according to yetanother embodiment of the present disclosure.

FIG. 16F is a schematic perspective view of two storage structuresarranged in a side by side relationship and depicting a mobile,manipulator robot traversing the lateral sides of the storagestructures.

FIG. 17 is a schematic illustration of an alternative pneumatic systemfor use with the mobile, manipulator robot of FIG. 9A or the mobile,manipulator robot of FIG. 16A.

FIG. 18 is a cross-section view of a modified gripping tool for use withthe alternative pneumatic system of FIG. 17.

FIG. 19 is a partial perspective view of a modified storage structureincluding a gantry frame supporting a robotic picking arm equipped witha pneumatic gripping tool.

FIG. 20 is a schematic illustration of another modified storagestructure including an assembly positioned above the storage structureand pneumatic supply lines extending from the assembly toward thestorage structure.

FIG. 21 is a flowchart illustrating an example method of using acomputing system to control the operation of the mobile, manipulatorrobot of FIG. 9A or the mobile, manipulator robot of FIG. 16A.

FIG. 22 is a flowchart illustrating an example method of using anoperating interface to control the operation of the mobile, manipulatorrobot of FIG. 9A or the mobile, manipulator robot of FIG. 16A.

FIG. 23 is a schematic perspective view illustrating a mobile,manipulator robot traversing a warehouse floor and picking inventoryitems from a shelf.

FIG. 24 is a schematic perspective view of the mobile, manipulator robotof FIG. 16A performing a digging operation.

FIG. 25 is a schematic top elevation view of a plurality of the mobile,manipulator robots of FIG. 16A including one or more container retrievaldevices.

FIGS. 26 and 27 are flowcharts illustrating example order fulfillmentprocesses.

DETAILED DESCRIPTION

As used herein, when terms of orientation, for example, “vertical” and“horizontal” or relative terms such as, “above,” “upwards,” “beneath,”“downwards” and the like are used to describe the orientation orrelative position of specific features of the storage structure ormobile, manipulator robot, the terms are in reference to the orientationor the relative position of the features in the normal gravitationalframe of reference when the storage structure is positioned with abottom of the storage structure resting on a surface. Also as usedherein, the terms “substantially,” “generally,” and “about” are intendedto mean that slight deviations from absolute are included within thescope of the term so modified.

FIGS. 1 and 2 illustrate a storage structure for efficiently storing aplurality of stackable containers 10, also known as bins, according tothe prior art. Containers 10 are stacked on top of one another to formstacks 12 and are arranged in a frame structure 14. Each container 10typically holds a plurality of product items (not shown). The productitems within each container 10 may be identical, or may be of differentproduct types.

Frame structure 14 includes a plurality of vertical members 16 thatsupport a first set of parallel horizontal members 18 extending in afirst direction (e.g., the X-direction), and a second set of parallelhorizontal members 20 extending in a second direction (e.g., theY-direction). Horizontal members 18 and horizontal members 20 form aplurality of horizontal grid spaces within which stacks 12 are housed.Frame structure 14 is thus constructed to guard against horizontalmovement of the stacks 12 of bins 10 and to guide vertical movement ofthe bins.

The uppermost level of frame structure 14 includes rails 22 arranged ina grid pattern across the top of horizontal members 18 and horizontalmembers 20. With additional reference to FIGS. 3A-3C and 4, rails 22support a plurality of robotic load handling devices 30. A first set ofparallel rails 22 a guides movement of load handling devices 30 in afirst direction (e.g., the X-direction) across the top of framestructure 14, and a second set of parallel rails 22 b, arrangedperpendicular to the first set of parallel rails, guides movement of theload handling devices in a second direction (e.g., the Y-direction)across the top of the frame structure. In this manner, rails 22 allowload handling devices 30 to move laterally in two directions (in theX-direction and in the Y-direction) across the top of frame structure 14so that the load handling devices can be moved into position above anyone of the stacks 12 of bins 10.

Each load handling device 30 includes a vehicle 32 with a first set ofwheels 34, consisting of a pair of wheels on the front of the vehicleand a pair of wheels on the back of the vehicle, arranged to engage withtwo adjacent rails of the first set of rails 22 a. Similarly, a secondset of wheels 36, consisting of a pair of wheels on each lateral side ofthe vehicle, is arranged to engage with two adjacent rails of the secondset of rails 22 b. Each set of wheels 34, 36 can be lifted and lowered,so that either the first set of wheels 34 or the second set of wheels 36is engaged with the respective set of rails 22 a, 22 b depending on thedesired direction of movement of vehicle 32.

When the first set of wheels 34 is engaged with the first set of rails22 a and the second set of wheels 36 is lifted clear from the second setof rails 22 b, the first set of wheels can be driven, by way of a drivemechanism (not shown) housed in vehicle 32, to move the load handlingdevice 30 in the X-direction. To move the load handling device 30 in theY-direction, the first set of wheels 34 is lifted clear of rails 22 a,and the second set of wheels 36 is lowered into engagement with thesecond set of rails 22 b. A drive mechanism (not shown) associated withthe second set of wheels 36 can then be used to drive the second set ofwheels in the Y-direction.

Load handling device 30 is also equipped with a crane device 40 having acantilever arm 42 that extends laterally from the top of vehicle 32. Agripper plate 44 is suspended from cantilever arm 42 by cables 46 thatare connected to a winding mechanism (not shown) housed within vehicle32. Cables 46 thus can be spooled into or out from cantilever arm 42 toadjust gripper plate 44 with respect to the vehicle 32 in theZ-direction.

Gripper plate 44 is adapted to engage with the top of a bin 10. Forexample, gripper plate 44 may include pins (not shown) that mate withcorresponding holes (not shown) in the rim that forms the top surface ofbin 10 and sliding clips (not shown) that are engageable with the rim togrip the bin. The clips are driven into engagement with bin 10 by asuitable drive mechanism housed within gripper plate 44, which may bepowered and controlled by signals carried through cables 46, or througha separate control cable (not shown).

To remove a bin 10 from the top of a stack 12, the load handling device30 is moved as necessary in the X and Y directions so that the gripperplate 44 is positioned above the stack in which the desired bin islocated. Gripper plate 44 is then lowered and brought into engagementwith the bin 10 on top of stack 12, as shown in FIG. 3C. After the clipshave engaged with and secured bin 10, gripper plate 44 and, in turn thebin, may then be pulled upwards by spooling cables 46. At the peak ofits vertical travel, bin 10 is accommodated beneath cantilever arm 42and is held above rails 22. In this way, load handling device 30 cantransport bin 10 to another location. Cables 46 are long enough to allowhandling device 30 to retrieve and place bins 10 at any depth withinstack 12, including the floor level. Vehicle 32 is sufficiently heavy tocounterbalance the weight of bin 10 and to remain stable during thelifting process. Much of the weight of vehicle 32 is attributed to thelarge and heavy batteries that are required to power and operate thedrive mechanisms of wheels 34, 36.

The known storage structure, as shown in FIG. 4, may include a pluralityof load handling devices 30 that operate simultaneously to increase thethroughput of the system. The storage structure depicted in FIG. 4includes two ports 24, or shafts, for transferring bins 10 into or outof the storage structure. An additional conveyor system (not shown) maybe associated with each port 24. In this manner, bins 10 that aretransported to port 24 by load handling device 30 can be subsequentlytransferred to a picking/sorting station (not shown) where the productscontained in the bins are picked and sorted into individual orders.Similarly, bins 10 can be moved by the conveyor system to port 24 froman external location, such as a bin-filling station (not shown), andtransported to a stack 12 by the load handling devices 30 to restock thestorage structure.

If it is necessary to retrieve a bin (“target bin”) that is not locatedon the top of stack 12, then the overlying bins 10 a (“non-target bins”)(e.g., the bins located between the target bin 10 b and rails 22) mustfirst be moved to allow load handling device 30 to access the targetbin. This operation is referred to as “digging”.

FIG. 5 illustrates a known digging operation in which one of the loadhandling devices 30 sequentially lifts each non-target bin 10 a from thestack 12 of bins 10 containing target bin 10 b. Each of the non-targetbins 10 a may be placed in a temporary location on top of another stack12. After each of the non-target bins 10 a have been removed, target bin10 b can be extracted from frame 14 by load handling device 30 andtransported to port 24. After target bin 10 b has been extracted,non-target bins 10 a may be placed back in the original stack 12 torestore the original order of the stack less the target bin.

Each of the load handling devices 30 may be operated under the controlof a central computer. Each individual bin 10 in the system is tracked,so that the appropriate bins can be retrieved, transported and replacedas necessary. For example, during a digging operation, the temporarylocations of each of the non-target bins 10 a is logged, so that thenon-target bins can be replaced in the stack in a particular order.

While the system illustrated in FIGS. 1-5 allows for the dense storageof products, it requires the transportation of entire containers ofproducts back-and-forth between the stacks and the picking/sortingzones, during which time products cannot be picked and sorted into newincoming orders, thus reducing total system throughput. In order tominimize bin transportation, target bins 10 b are typically onlyretrieved and transported to the picking/sorting stations after multipleorders have been placed for a product item of one type. Although thismethod reduces bin transportation, order fulfilment times are oftenlengthier than desired, particularly if an order contains one or moreproducts that are infrequently ordered by consumers. For this reason,“piece picking” inventory from the known frame structure 14 has beencontemplated. U.S. Pat. Pub. Nos. 2018/0319590 and 2018/0346243, forexample, disclose a robot equipped with a picking arm to pick individualitems from a container located in the frame structure. Nevertheless, thepicking robots and systems disclosed in U.S. Pat. Pub. Nos. 2018/0319590and 2018/0346243 are not robust enough to handle the picking of a widevariety of products.

The present disclosure, on the other hand, provides a robot having apicking manipulator (sometimes referred to herein as a “picking arm”)coupleable to a gripping tool for grasping a variety of products andplacing the products into one of a plurality of order containers. Todate, a major barrier in developing robotic picking arms has been theinability of the picking arm to consistently grasp products of varyingsizes, shapes, weights, materials, surface textures, densities, massdistributions, stiffnesses and fragilities. While picking arms equippedwith pneumatic gripping tools have been contemplated as one potentialsolution for gripping a wide variety of products, these gripping toolsrequire extensive suction force and flow rate that can only be producedby large vacuum pumps and/or compressors (e.g., smaller vacuumpumps/compressors are only capable of providing adequate suction for avery small range of items). Oversized pneumatic compressors and/orvacuum pumps, however, are prohibitively large for load handling device30 or similarly sized vehicles. In other words, load handling device 30is not capable of carrying a large pneumatic compressor and/or vacuumpump within vehicle body 32. Increasing the size of the vehicle body 32to allow load handling device 30 to carry an oversized pneumaticcompressor and/or vacuum pump would require modifying the footprint ofthe vehicle body to a size that would consume a large number of gridspaces. As a result, fewer load handling devices would be able occupythe grid at a single time and throughput of the system would be reduced.For this reason, robots with pneumatic gripping tools have generallybeen confined to the floor of a warehouse and are often fixed to astationary base.

The present disclosure provides a robotic system including a storagestructure equipped with a pneumatic air supply system and a compactmobile, manipulator robot selectively coupleable to the pneumatic airsupply system to allow the mobile, manipulator robot to grasp inventoryitems with its pneumatic gripping tool. As a result, the robot can graspa large variety of products while traversing across the storagestructure and support larger payloads during grasping. The ability ofthe mobile, manipulator robot to quickly and efficiently grasp a widevariety of inventory items is further improved by the robots ability toquickly switch between two or more pneumatic gripping tools and requestgrasping assistance from a teleoperator if the robot is unable toautonomously grasp an item during an edge case scenario (or thepredicted control instructions have high uncertainty or low confidence).The mobile, manipulator robot can thus to continue its normal operationwith minimal downtime or interruption. These improvements, among otheradvantages, are discussed in further detail in this disclosure.

FIG. 6A is a schematic illustration of a robotic system 100 according toan embodiment of the present disclosure. A robot, such as mobile,manipulator robot 200 (sometimes referred to herein as “manipulatorrobot” or “robot”), may be housed in a storage system 101 such as awarehouse, or other fulfillment center (hereinafter “warehouse”), andtasked with picking inventory items contained within storage structure114. Robot 200 may operate in one of two modes: an autonomous mode, byexecuting autonomous control instructions, or a tele-operated mode, inwhich the control instructions are manually piloted (e.g., directlycontrolled) by an operator. While the term “control instructions”(whether autonomous or piloted) is primarily described herein asinstructions for grasping an item, it will be appreciated that the termmay additionally refer to a variety of other robotic tasks such as therecognition of an inventory item, the placement or release of a graspeditem (e.g., in a particular location or orientation) or any otherrobotic task that facilitates order fulfillment. In one embodiment,robot 200 may be a machine learning robot capable of executingautonomous or piloted control instructions.

Robotic system 100 includes one or more operator interfaces 102, atleast one of which may be located at a remote site outside of warehouse101, one or more processor-based computer systems 103, each of which arecommunicatively coupled via one or more network or non-networkcommunication channels 104, and one or more storage devices 105, whichstore, for example, a machine learning grasp pose prediction algorithmused to predict grasping poses for manipulator robot 200 to execute andgrasp inventory items. While storage device 105 is illustrated as beingseparate from computer system 103, in at least some implementations, thestorage devices can be an integral part or component of the computersystem (e.g., memory such as RAM, ROM, FLASH, registers; hard diskdrives, solid state drives). As used herein, the terms “remoteprocessor” or “remote computer” refer to a processor in communicationwith and located remote from the hardware of the referenced robot andmay include, for example, one or more processors or a single centralprocessor for coordinating and automating fulfillment tasks between therobots. On the other hand, when the term “onboard” is used herein, theterm means that the component is being carried by the referenced robot.For example, an “onboard processor” means that the processor is locatedwithin the hardware of the referenced robot. When the general term“processor” or “computer” is used herein, the term may refer to anyremote processor, any on-board processor or a combination of the same,unless explicitly indicated otherwise.

Operator interface 102 includes one or more input devices to capturecontrol instructions from an operator and one or more output devices.The one or more user interface devices 102 may be, for example, apersonal computer, a tablet, (smart) phone, a wearable computer, and thelike. Exemplary input devices include keyboards, mice, touch screendisplays, displays (e.g., LCD or OLED screen), controllers, joysticksand the like. In this regard, a teleoperator may input synchronous(real-time) or asynchronous (scheduled or queued) control instructionswhich may be, for example, click point control instructions, 3d mousecontrol instructions, click drag control instructions, keyboard or arrowkey control instructions, and/or image captured hand or body controlinstructions. Exemplary output devices include, without limitation,displays (e.g., LCD or OLED screen), head mounted displays, speakers,and/or haptic feedback controllers (e.g., vibration element,piezo-electric actuator, rumble, kinesthetic, rumble motor). Operatorinterface 102 thus may be utilized by an operator to observe roboticpicking, for example, aspects of manipulator robot 200 and/or theinventory stored within storage structure 114. Operator(s) may view orsee a representation of manipulator robot 200 performing one or moretasks such as grasping an item by reviewing one or more still and/ormoving images of the manipulator robot and/or its environment. Theseimages and/or video may be replayed and/or viewed in real time. Ifmanipulator robot 200 is unsuccessful at autonomously performing thetask, the operator can utilize operator interface 102 to instruct therobot to grasp a product item and/or release the product item into adesired order container. Although operator interface 102 is primarilydescribed herein in connection with assisting robot 200 in performinggrasping tasks, it will be appreciated that the interface may be used atany time (including prior to a failed grasping attempt) to allow ateleoperator to manually control the robot and to perform anymanipulation task including the picking, rearranging, packing orrepackaging of one or more items, picking up dropped items, manipulatingitems in inventory bins or any other order fulfillment tasks includingthe performance of inventory audits, replenishment tasks, systeminspections, product identification and/or to override other autonomouscontrol instructions.

Computer system 103 coordinates the operation of robotic system 100.Computer system 103 can be a processor based computer system. Theprocessor may be any logic processing unit, such as one or moremicroprocessors, central processing units (CPUs), digital signalprocessors (DSPs), graphics processing units (GPUs),application-specific integrated circuits (ASICs), programmable gatearrays (PGAs), programmed logic units (PLUs), and the like. In someimplementations, computer system 103 may include a control subsystemincluding at least one processor.

Examples of a suitable network or non-network communication channels 104include a wire based network or non-network communication channels,optical based network or non-network communication channels, wireless(i.e., radio and/or microwave frequency) network or non-networkcommunication channels, or a combination of wired, optical, and/orwireless networks or non-network communication channels.

Mobile, manipulator robot 200 includes an interface to send and/orreceive processor readable data or processor executable instructions viacommunication channels 104 to computer 103. In this manner, computer 103can predict grasping poses (e.g., position and/or orientation and/orposture of the robotic picking arm) and send control instructions tomanipulator robot 200 to execute the predicted grasping pose and graspthe product item. If the control instructions are unsuccessful inperforming a task (e.g., grasping the item), or the remote computerdetermines that the predicted control instructions are unlikely to besuccessful, the system can automatically request intervention from theoperator, allowing robot 200 to be teleoperatively controlled from alocal or remote location.

As will be described in greater detail hereinafter, the present systemallows a teleoperator to remotely pilot manipulator robot 200 and movethe robot into a variety of grasping (or manipulation) poses to trainthe machine learning system to more accurately predict future autonomousrobot control instructions.

Although FIG. 6A illustrates two robots 200 located within a singlewarehouse, it will be appreciated that the system can include a singlerobot or any number of robots located within a single warehouse, or oneor more robots located within a plurality of warehouses. The roboticsystem is thus advantageously configured to allow one or more operatorsto teleoperatively pilot or control a plurality of manipulator robots200, via one or more operator interfaces 102, from a site located localor remote to the warehouses in which the robots are contained.

Storage structure 114, as shown in FIG. 6B, is configured to efficientlystore stackable containers 110, also referred to herein as bins. Eachbin 110 is configured to hold a plurality of product items (not shown)which may be identical, or of a variety of product types. Exampleproduct types include household items, apparel, consumer electronics,beauty products, groceries or any other product that may be stored andshipped from a warehouse. The products may be arranged within storagestructure 114 in a number of ways to optimize picking/packing. Forexample, the products may be arranged based on product type (e.g.,similar products are grouped together), the speed in which products needto be fulfilled, the environment (e.g., temperature) the items need tobe stored within, the number of times a product is traditionally sold ina given time period, items size, items that are commonly purchasedtogether, etc.

In situations where the product requires specific storage conditions(e.g., temperature or humidity) such as groceries, containers 110 may bepacked with dry ice a similar mechanism to regulate the storageconditions of the specific product type. Alternatively, storagestructure 114 may be constructed to include one or more isolated andinsulated refrigeration or freezer areas. Each refrigeration or freezerarea may rely on cryogenic cooling to achieve a desired temperature, ormay alternatively utilize a separate refrigeration system formed, forexample, of a condenser, a compressor and an evaporator configured tocycle gas through the system to refrigerate and/or freeze the insulatedarea. Product items such as groceries may be stored in containers 110and arranged within storage structure 114 in either one of the frozenarea, refrigerated area, and/or at room temperature based upon thestorage requirements of the product type. In some instances, thesefreezer/refrigerated areas may be located on the lower levels of storagestructure 114. Grocery products may be naturally slotted closer to orfurther from the frozen and refrigerated areas based upon theirindividual temperature and storage climate requirements. Thisconfiguration also isolates the robots positioned on top of storagestructure 114 from the freezer/refrigerated areas. Nevertheless, shouldthe robots, or a portion thereof, need to access the freezer orrefrigerated area, the robot may include a heating component to regulatethe temperature of its electronics and other systems.

Containers 110 preferably have an open end through which the productscan be retrieved. The open end of container 110 may be an open top or anopen lateral side. The bottom of containers 110 may have an inwardlytapered interior surface that facilitates the rolling and/or the slidingof inventory products toward the center of the container and away fromthe sidewalls of the container to facilitate picking. In some cases, thebottom of containers 110 may include slidable, pivotable or bomb baydoors to facilitate the dumping of inventory items from the container toother containers or elsewhere. The bottom of containers 110 may also bedesigned to nest within or against a rim that forms the upper surface ofanother container to prevent the containers from moving laterallyrelative to one another when stacked. Thus, storage structure 114 neednot include any, or significantly less, support members than counterpartframe structure 14. As a result, storage structure 114 may cost less tomanufacture and may be installed more quickly than frame structure 14.

Storage structure 114 may nevertheless include vertical members 116 thatsupport a first set of horizontal members 118 extending in a firstdirection (e.g., the X-direction) and a second set of horizontal members120 extending in a second direction (e.g., the Y-direction). Horizontalmembers 118 and horizontal members 120 form a plurality of horizontalspaces for housing stacks 112. The horizontal spaces are constructed toguard against lateral movement of the stacks of bins 110. Storagestructure 114 may additionally include one or more ports 121 or shaftsto transfer bins into or out of the storage structure. A conveyor beltor shuttle system (not shown) may be associated with each port 121 totransport bins 110 to an external location. For example, a bincontaining products for shipment may be transported down port 121 to anexternal location for further packaging and/or shipment, while an emptybin may be transported down the port to a bin-filling station (notshown) for replenishment and then subsequently transported up the portand to one of the stacks 112 to restock the storage structure.

The uppermost level of storage structure 114 may include a first set ofrails 122 extending in a first direction (e.g., X-direction) and/or asecond set of rails 124 extending in a second direction (e.g.,Y-direction). In embodiments in which storage structure 114 includes thefirst set of rails 122 and the second set of rails 124, the combinationof the first and second set of rails forms a horizontally oriented grid126 having a plurality of grid spaces 127. Rails 122, 124 allow one ormore robots to move about the grid 126 above the stacks 112 of bins 110.At least one of the vertical members 116, horizontal members 118,horizontal members 120 or rails 122, 124 may define a channel thattransports fluid such as compressed air to the robots installed on grid126 as is discussed in further detail hereinafter.

As shown in FIG. 6C, a plurality of similarly constructed storagestructures 114 with shallower stacks (e.g., fewer containers per stack)may be layered on top of one another to reduce the time it takes to diga target container 110 b (e.g., the container storing a desiredproduct), which in turn, increases the throughput of the system. In suchscenarios, each storage structure 114, or level, would be spaced apartfrom an adjacent level with enough clearance between each level to allowone or more robots to move about a respective grid 126. One or moreelevators and/or ramps having inclined and/or declined rails 129 (in theZ-direction) may be provided between the grids 126 of adjacent storagestructures 114 to allow the robots to migrate between the levels asdesired.

Referring to FIG. 6B, one or more of the lateral sides of storagestructure 114 may additionally or alternatively include the second setof rails 124 extending in the second direction (e.g., Y-direction)and/or a third set of rails 125 extending in a third direction (e.g.,Z-direction). In embodiments in which storage structure 114 includes thesecond set of rails 124 and the third set of rails 125, the combinationof the second and third set of rails 124, 125 forms a verticallyoriented grid 126 having a plurality of grid spaces 127. Manipulatorrobot 200 may traverse vertical grid 126, extract bins 110, and pickfrom the extracted bins housed in shelving, racks or stacks on thelateral sides of storage structure 114. When the term “grid” is usedherein without an orientation qualifier (e.g., vertical or horizontal),the term may refer to any grid structure formed by a combination ofrails 122, 124, 125, whether the grid be horizontally oriented orvertically oriented.

It is also envisioned that a plurality of similarly constructed storagestructures 114 may be positioned laterally adjacent to one another (notshown), to increase storage capacity. In such scenarios, each storagestructure 114 would be spaced apart from an adjacent storage structurewith enough space between the adjacent storage structures to allow arobot 200 to traverse about a respective vertically oriented grid 126and access containers 110 housed within either of the adjacent storagestructures.

Referring to FIGS. 7A and 7B, each one of the rails 122, 124, 125forming grid 126 may be extruded or otherwise formed from a highlyconductive metal such as aluminum. A power source P may be coupled togrid 126 to supply a voltage to rails 122, 124, 125 and, in turn, toselectively provide a voltage to robot 200 to recharge small batteriesor super/ultra-capacitors of the robot and/or directly power the variousdrive mechanisms of the robot. The power may be transferred from grid126 to robot 200 in one of several methods. For example, grid 126 mayhave a single polarity such as a negative charge, while a structure orceiling above the grid (not shown) is positively charged (or viceversa). In this embodiment, robots 200 may include an antenna 219 (shownin FIG. 9A) which contacts the positively charged structure or ceilingabove grid 126 and completes the circuit between the oppositepolarities. In an alternative arrangement, adjacent rails of one set ofthe parallel rails 122, 124 and/or 125 may have opposite polarities suchthat when robot 200 is disposed on the adjacent parallel rails,conductive brushes (e.g., contact elements) of the robot will completethe circuit. For example, a first one of the parallel rails 122 may havea positive polarity while an adjacent one of the parallel rails 122 mayhave a negative polarity. In this manner, robot 200 need not include thelarge onboard batteries associated with load handling device 30. As aresult, robot 200 is less bulky and more maneuverable than its loadhandling device 30 counterpart.

Rails 122, 124, 125 may include a double u-channel or profiled trackhaving an upper surface 128, outer surfaces 130, inner surfaces 132 anddrive surfaces 136 a, 136 b (collectively “drive surfaces 136”). In thismanner, two robots may traverse a single rail 122, 124, 125, increasingthe number of robots capable of driving on grid 126 at any given time.For example, a first robot supported on drive surface 136 a may pass asecond robot supported by drive surface 136 b. The upper surface 128,outer surfaces 130 and inner surfaces 132 of rails 122, 124, 125 may beanodized or painted with a non-conductive coating to prevent the robotsor storage structure 114 from short circuiting and to minimize the riskof electrocution. In other words, the drive surfaces 136 of rails 122,124, 125 may be the only surfaces of the rails that remain at leastpartially or entirely electrically charged (aside from the terminalends, or a small section of the terminal ends of the rails, which arenot anodized for the purpose of transmitting power along the rails ofthe grid).

Storage structure 114 further includes a fluid supply system 138configured to supply fluid such as compressed air to robot 200 when therobot is installed on rails 122, 124, 125. Fluid supply system 138 thuseliminates the need for robot 200 to carry a bulky onboard aircompressor or vacuum generator to operate its pneumatic gripping tool248 (FIG. 9A) and grasp inventory items stored in containers 110. Fluidsupply system 138 includes a fluid source S and a supply line 140. Fluidsource S may be a compressor, such as a pneumatic compressor, to supplycompressed air to supply line 140. Alternatively, fluid source S may bea vacuum pump or vacuum generator.

While supply line 140 is primarily described and illustrated herein asextending through the rails 122, 124, 125 of grid 126, it will beappreciated that the supply line may alternatively be formed by orextend at least partially through the channels of vertical members 116,horizontal members 118 or horizontal members 120 forming the frame ofstorage structure 114, attached to or otherwise coupled to an externalsurface of the rails and/or the frame structure, or otherwise be inclose proximity of the rails so long as the fluid supply is accessibleto manipulator robot 200 when the robot is positioned on the grid.

As shown in FIG. 7A, supply line 140 may include a series of channels142, conduits 144 and ports 146. Channels 142 may extend along an entirelength of rails 122, 124, 125, and are preferably, embedded within alower portion of the u-channel such that the channels extendcontinuously in a longitudinal direction of a respective rail withoutinterruption at the intersections of rails 122 and rails 124, or theintersection of rails 124 and 125. A plurality of conduits 144 mayextend between channel 142 and a port 146 located at a surface of arespect rail. In a preferred embodiment, at least one of rails 122, 124,125 that surrounds each one of grid spaces 127 has a conduit 144. Grid126 is thus capable of supplying fluid such as compressed air to robot200 irrespective of the robot's location on the grid.

Referring to FIGS. 8A and 8B, a plurality of valves 150 may be disposedwithin supply line 140, for example, within the conduits 144 of rails122, 124, 125 or within channels formed by vertical members 116,horizontal members 118, and/or horizontal members 120. Each valve 150 istransitionable between a closed condition in which the compressed air iscontained within supply line 140 and an open condition in which thesupply line is in fluid communication with the environment such thatcompressed air may be supplied to manipulator robot 200. Each valve 150may include a biasing member 152, such as a spring, and a plug 154coupled to the spring to seal port 146. When spring 152 is in a neutralor unbiased condition, the spring 152 biases the plug into the port 146,which seals the compressed air within supply line 140. Alternativevalves may be used to seal compressed air within supply line 140. Forexample, the valve may be constructed as any passively or activelyactuated valve capable of being transitioned between a closed conditionand an open condition, such as an electrohydraulic servo valve.

With specific reference to FIG. 8B, the rails 122, 124, 125 of grid 126may define a cavity 143 aligned with a longitudinal axis of conduit 144.Cavity 143 may include a tapered edge extending from the upper surface128 of rails 122, 124, 125 toward port 146. A magnet 157, or otherferrous material, may surround port 146 to magnetically couple robot 200to grid 126 during the transference of the compressed air from supplyline 140 to robot 200. A gasket, such as an O-ring 155, may be providedaround port 146 to seal the connection between robot 200 and grid 126,and/or at any other location surrounding the valves 150 to preventcompressed air from leaking out of supply line 140. The compressed airof supply system 138 may be selectively accessed by mobile, manipulatorrobot 200 to provide the necessary suction to allow the manipulatorrobot to piece-pick inventory items ranging in sizes, shapes, weights,materials, surface textures, densities, mass distributions, stiffnessesand fragilities.

Referring to FIGS. 9A and 9B, manipulator robot 200 includes a vehiclebody 202, a mobility assembly 204 configured to guide movement of thevehicle body along rails 122, 124, 125 and a picking manipulator 206also referred to herein as a “picking arm”. Manipulator robot 200 alsoincludes a communication interface to send and receive data between themanipulator robot and remote computer 103 and/or the manipulator robotand operator interface 102. The data may include information obtainedfrom a positioning sensor and relate to the position of the manipulatorrobot relative to storage structure 114 or the warehouse 101 in generalto enable remote computer 103 to control movement of the robot aboutgrid 126 or about the warehouse. The position sensor may be a globalpositioning system, a local/indoor positioning system, a local featurepositioning system or a combination thereof. The global positioningsystem may be a GPS system. The local or indoor positioning system maybe an indoor positioning system using different technologies, includinglight, radio waves, magnetic fields or acoustic signals to measure adistance or time of flight to nearby anchor nodes (nodes with knownfixed positions such as WiFi/LiFi access points, Bluetooth beacons orUltra-Wideband beacons, magnetic positioning, or dead reckoning) toactively locate mobile devices or tags to provide ambient location orenvironmental context. The local feature positioning system on the otherhand may utilize conductive, capacitive, infrared (IR) or other sensorsused to detect features within the warehouse, for example, a sensor todetect and count rail or grid space crossings, a magnetic sensordesigned to detect magnets or ferrous material in grid 126, an imager toread barcodes or AR/QR codes on bins 110, the rails or other structures(which can subsequently be relayed to the remote processor 103 todetermine the location of the mobile manipulator robot), an imagercapable of performing simultaneous localization and mapping (SLAM),encoders in the mobility assembly 204 to measure distances traveled,magnetic, NFC, RFID, or any other type of positioning sensor within anyof the mobile robots described herein and/or the grid so long as theremote computer can determine the location of each individual mobilerobot and control the position of each individual mobile robot. The datamay also include data obtained from a sensor relating to the inventory(hereinafter “Inventory Data”) (e.g., location, dimensions, shapes,weights, materials, porosities, surface textures, colors, densities,mass distributions, stiffnesses, fragilities or the like) that assistthe computer or a teleoperator in distinguishing between differentproducts located in the container and/or predicting a grasping pose forgrasping the product item.

Vehicle body 202 may be formed of four sidewalls 208 a, 208 b, 208 c,208 d (collectively “sidewalls 208”), an open bottom end 210 and an opentop end 212. The sidewalls 208 are preferably sized such that vehiclebody 202 has a footprint of a single grid space 127. In other words,when robot 200 is positioned on the horizontal grid 126, two opposingsidewalls (e.g., 208 a, 208 c) are positioned over two adjacent rails122 extending in the X-direction, while the other two opposing sidewalls(e.g., 208 b, 208 d) are positioned over two adjacent rails 124extending in the Y-direction. In other embodiments, the vehicle body 202of robot 200 may have a footprint that is larger than a single gridspace 127. The open bottom end 210 and the open top end 212 of vehiclebody 202 allow picking arm 206 to extend through the vehicle body andgrasp a product contained in a target bin 110 b, which may be locateddirectly beneath the body (e.g., the bin located on the top of the stackof bins aligned with the vehicle body in the Z-direction). Picking arm206 may alternatively be used to pick products contained in target binslocated laterally adjacent to the vehicle body 202 as shown in FIG. 9A.

One or more of the sidewalls 208 of vehicle body 202 may optionallyinclude a pivotable digging plate (not shown) for digging into a stack112 and pulling a target bin to the top of a particular stack and/or fortransporting bins for replenishment purposes. The digging plate may bepivotable between a collapsed condition in which the digging plate liesflush against a respective interior or exterior surface of one of thesidewalls 208 of vehicle body 202 and an operating condition in whichthe digging plate extends radially away from and perpendicular to therespective sidewall of the vehicle body. The digging plate may besimilar to gripper plate 44 of load handling device 30 in that thedigging plate is configured to be lowered in the Z-direction and broughtinto engagement with any of the bins 110 located in stack 112. Likegripper plate 44, the digging plate may be adapted to pull bins 110upwards by spooling cables, which are long enough to retrieve a targetbin located at any depth within stack 112. However, robot 200 need notinclude a digging plate or another mechanism for digging the containersfrom stack 112. System 100 could instead rely on the combination ofmanipulator robot 200 and a separate robot specifically adapted toperform digging tasks. The digging robot may be known load handlingdevice 30 or digging robot 205 (FIGS. 6C and 6D).

With specific reference to FIG. 6D, digging robot 205 may include avehicle body having a container receiving cavity and a digger 207extendable beneath the body. Digger 207 may be a scissor lift or includea series of telescoping beams or other compact linear actuators withlong stroke. In this manner, digger 207 may reach beneath grid 126 tolift a single container 110, or a plurality of containers (e.g., atarget bin 110 b and each of the non-target bins 110 a overlying thetarget bin) through the receiving cavity and above the grid in a singlelift. Alternatively, digger 207 may be positioned on a single externalside of digging robot 200 and include a latching device such as a hookfor engaging with one or more lateral sides of containers 110. In thismanner, digging robot 205 may reach beneath grid 126 to lift a singlecontainer 110, or a plurality of containers (e.g., a target bin 110 band each of the non-target bins 110 a overlying the target bin) abovethe grid and on a lateral side of the digging robot (e.g., withoutlifting the containers through the container receiving cavity of thedigging robot). The digger 207 of digging robot 205 may be electrically,pneumatically or otherwise actuated.

The internal surface of the sidewalls 208 of robot 200 may also includea latch, hook, digging plate or other mechanism (not shown) for couplingorder bins 214 a, 214 b (collectively “order bins 214”) within thevehicle body 202 of the manipulator robot such that the combination ofthe piece picking robot and the one or more order bins have a footprintof approximately one grid space 127. The latch, hook, digging plate orother mechanism may alternatively be placed on an external surface ofone or more of the sidewalls 208 of vehicle body 202 to couple one ormore order bins 214 around the vehicle body as shown in FIGS. 9A and 9B.

Each of the order bins 214 may correspond to one or more orders. If asingle order bin corresponds to more than one order, the bin may bepartitioned to separate the multiple orders in a single bin, or remainun-partitioned with all of the items from multiple orders mixedtogether. For example, order bin 214 a may correspond to a firstconsumer's order and order bin 214 b may correspond to a secondconsumer's order. Thus, after robot 200 has picked a product from targetbin 110 b, the product may be placed directly into the order bincorresponding to the order of the consumer who purchased the product. Inone embodiment, the bottom end of order bins 214 may include slidable,pivotable or bomb bay doors to facilitate the dumping of items intoother containers, areas, or down ports 121 for further sorting orprocessing. It will be appreciated, however, that piece picking robot200 need not carry any order bins 214. Instead, piece picking robot 200may be used only for grasping products, which may be subsequently placedinto order bins 214 carried by a “transporting robot” (e.g., a robottasked with carrying around order bins) (not shown). In this manner,both manipulator robot 200 and the transporting robot may move alonggrid 126 and meet at certain picking or transfer locations.

With specific reference to FIG. 9B, robot 200 further includes one ormore sensors 262 such as an RGB or RGB-D camera, video recorder, LightDetection and Ranging (LIDAR), and the like, oriented to capturepictures, point clouds, video etc. (generally referred to herein as “animage” or “images”) of the product item(s) stored within containers 110.Although, sensor 262 is illustrated as being coupled to picking arm 206,the sensor may alternatively be coupled to the body 202 of robot 200 orto gripping tool 248 (shown in FIG. 12B). The image(s) may betransmitted via network or non-network communication channels 104 toprocessor 103 which, in some instances, may additionally be relayed tooperator interface 102. In this manner, processor 103 may implicitly orexplicitly analyze the images and then execute a machine learningalgorithm, located within storage device 105, to predict a grasping poseto grasp the desired product item, before transmitting the grasping posecontrol instructions to robot 200 via communication channels 104 which,when executed by the robot, causes the picking arm 206 of the robot toapproach and attempt to grasp the item. Although the grasping pose canrefer to a single pose, grasping an item often requires a set ofconsecutively run poses. As used herein, the term “grasping pose” mayrefer to a single pose or a set of consecutively run poses. The imagesare preferably continuously captured as robot 200 traverses grid 126 andtransmitted to remote computer 103. In this manner, remote computer 103may determine a grasping pose for the picking arm 206 of robot 200, orthe picking arm of another manipulator robot, before the manipulatorrobot reaches the picking position, thus increasing throughput ofrobotic system 100.

FIG. 9C is a flow chart showing a method 400 of autonomously determininga grasping pose. The process for determining a grasping pose may begin,at block 402, with a command from processor 103 that instructs sensors262 to capture an image of the inventory disposed within a targetcontainer 110 b.

The image(s) may then be transmitted, at block 404, over network ornon-network communication channels 104 to processor 103. Upon receipt ofthe image, processor 103 may analyze the images and the Inventory Dataof the items stored within the target container 110 b at block 406.

Based on the Inventory Data, processor 103 may execute one or moregrasping pose detection algorithms (which can be neural networks ormachine learning algorithms stored on storage device 105) to predict oneor more grasping pose candidates at block 408. Processor 103 may thenimplement a policy, at block 410, which utilizes one or more metrics,checks and filters to select one or more of the predicted grasping posecandidates for robot 200 to execute sequentially or to add to its queue.Then, at block 412, processor 103 produces, makes, or generates a signalincluding processor readable information that represents the selectedgrasping pose and sends the signal through communication channels 104 torobot 200. It will be appreciated, however, that robot 200 canalternatively run part of, or the entirety of, the grasping model on anonboard computer rather than relying on remote computing andcommunications.

As shown in FIGS. 9D and 9E, sensors 262 and the grasping model may workin concert to identify a grasping region 414 of product items, definedas a specific area on the product item or packaging of the product itemas a whole that manipulator robot 200 has a high likelihood ofsuccessfully grasping. Grasping region 414 may be a relativelynon-porous and flat surfaced region of the product item and/or theproduct packaging when gripping tool 248 utilizes a suction force,antipodal surfaces when the gripping tool includes finger-like graspingelements, or a non-flat surface or edge when the gripping tool is auniversal jamming gripper, or any other geometric properties conduciveto being handled by a specific type of gripper capable of picking andhandling items with specific geometric, material, and surfaceproperties. FIG. 9D illustrates product items of different types withina target container 110 b. FIG. 9E illustrates the identification of agrasping region 414 of the product items located within an area of thetarget container.

Referring to FIG. 10A, mobility assembly 204 is configured to guidemovement of vehicle body 202 along rails 122, 124, 125 and positionrobot 200 over, or laterally adjacent to a target bin 110 b (e.g., a bincontaining the product to be picked). Mobility assembly 204 may includea plurality of wheels 216, a motor 218 and one or more transmissions(belts or linkages) 220 operably coupling each one of the wheels to themotor. Wheels 216 may be configured with a smooth outer surface (e.g.,cylindrical, disk, or spherical) or as a gear and may be formed of anymaterial such as rubber, metal or plastic so long as the wheels canguide movement of vehicle body 202 along rails 122, 124, 125 to positionrobot 200.

Each one of wheels 216 may include a direct drive (not shown) orquasi-direct drive (not shown) actuator within a hub with a magneticencoder, a hub motor (not shown) and a gear drive actuator (not shown)or a belt drive actuator (not shown) to rotate wheels 216 and movevehicle body 202 along the rails 122, 124, 125 in which the wheels arepositioned. Mobility assembly 204 may include four wheels 216, with onewheel being located at or adjacent to each one of the corners of vehiclebody 202. The orientation of wheels 216 is controlled by motor 218 andtransmission 220. More specifically, transmission 220 couples motor 218to each one of wheels 216 directly or indirectly such that rotation ofthe motor simultaneously rotates/pivots the orientation of each one ofthe wheels 216 between a first orientation in which each of the wheelsare oriented, for example, along rail 122, and a second orientation inwhich the wheels are aligned with rail 124 (e.g., 90 degrees). The fourwheels 216 can thus be used to guide movement of vehicle body 202 in twodirections, for example, along rails 122 (e.g., X-direction) and alongrails 124 (e.g., Y-direction). Transmission 220 can also simultaneouslypivot wheels 216 less than 90 degrees, or greater than 90 degrees, toorient the wheels and precisely control movement of robot 200 in anydirection when the robot is not positioned on grid 126. Consequently,robot 200 need not include a second set of wheels or a separate drivemechanism for lifting and disengaging the second set of wheels each timethe robot drives along a different rail, as is the case with known loadhandling device 30. Nevertheless, it will be appreciated that robot 200may alternatively be constructed with two separate sets of wheels anddrive mechanisms as described above with respect to load handling device30. In one embodiment, wheels 216 may include magnets or electro magnetsconfigured to act in concert with magnets or electro magnets in rails122, 124, 125 to slightly levitate and propel the robot along the rails.

Manipulator robot 200 may further include a prop mechanism 237, as shownin FIG. 10B, connected to the body 202 and used to prop the mobilityassembly 204 off of a driving surface. Prop mechanism 237 includes oneor more linear or rotary actuators 239 designed to move one or morestands 241 in the z-direction relative to the body 202 of robot 200.Linear actuator 239 includes a housing 243 coupled to the body 202,preferably on an internal surface of one or more of the sidewalls 208and a plunger 245 that is retractable toward the housing and extendableaway from the housing. Plunger 245 is connected to stand 241 which mayextend continuously or discontinuously around an inner surface ofsidewalls 208 adjacent to the lower end 210 of body 202. When plunger245 extends away from housing 243, stand 241 is moved downward and intocontact with a drive surface such as the rails of grid 126 and positionsthe stand beneath that of wheels 216 which, in turn, transfers the loadof manipulator robot 200 from the wheels to the stand. In this regard,the wheels 216 are suspended or floating above the drive surface so theorientation of the wheels can quickly and effortlessly be pivoted bymotor 218 and transmission 220 as explained above. The plunger 245 maythen be retracted to lift stand 241 away from the drive surface which,in turn, causes the wheels to re-engage the drive surface so thatmanipulator robot may then be moved.

The mobility assembly 204, or body 202, of manipulator robot 200 mayfurther include one or more electrical brushes or conductive elements221 (shown in FIG. 14B) to engage the inner drive surfaces 136 a, 136 bof rails 122, 124, 125 and transfer the charge from the rails to arelatively small onboard battery or super/ultra-capacitor and, in turn,to the drive motor or gear drive actuator of the robot. As a result,robot 200 may charge its battery or super/ultra-capacitor while therobot traverses grid 126. The throughput of the system is thus increasedbecause robot 200 need not be removed from grid 126 and/or paused inorder to charge or swap its battery or super/ultra-capacitor. Therelatively small onboard battery or super/ultra-capacitor also allowsrobot 200 to be lighter, faster and safer than its load handling device30 counterpart. Moreover, the small battery or super/ultra-capacitor maytemporarily power the drive motor and/or gear drive actuator to drivewheels 216 even when robot 200 is removed from and driven off of grid126. For example, robot 200 may be driven on the warehouse floor in anydirection to navigate the robot between grids 126 and/or to other areasof the warehouse so that the robot may assist with other fulfillmenttasks such as replenishment, picking/sorting inventory from shelving ora container (e.g., a bin, a tote, or any other structure holdinginventory), for example, at a picking/sorting station, and/or packagingthe picked/sorted inventory.

Referring to FIG. 11, robot 200 further includes a pneumatic coupler 222adapted to receive a fluid, such as compressed air, from fluid supplysystem 138. Coupler 222 is preferably extendable from a position withinthe sidewall 208 of vehicle body 202 to a position outside of thesidewall of the vehicle body in a manner that allows the coupler toselectively engage and disengage with valve 150. When coupler 222 ispositioned within the vehicle body 202 of robot 200, the coupler willnot interfere with other robots positioned on grid 126 or otherstructural features of the storage system. Coupler 222 may be agenerally hollow tube sized to be positioned within the cavity 143 ofrails 122, 124, 125. The mating end of coupler 222 may be tapered and/orinclude a self-alignment or misalignment handling device to assist inpositioning coupler 222 within cavity 143. The mating end of coupler 222may also include an O-ring (not shown), a magnet 223 for magneticallyengaging the magnet 157 or ferrous material disposed around port 146,and a device 224 for transitioning valve 150 between the closed and openconditions. Device 224 may be, for example, a mechanical member adaptedto push plug 154 into conduit 144 (away from port 146), or any otherdevice for electrically, magnetically, mechanically or otherwisetransitioning valve 150, or another valve, between the closed and openconditions. For example, a similarly constructed coupler may include oneor more conductive pads to provide power and actuate an electrohydraulicservo valve.

Robot 200 may optionally carry a small air tank 266 (FIG. 9A) forstoring compressed air. In some embodiments, air tank 266 may be smallerthan 20 cubic feet. The air tank 266 of robot 200 is in selectivecommunication with coupler 222. In this manner, robot 200 need notaccess the compressed air of supply system 138 each time the robotdesires to grasp a product item. Robot 200 may instead rely on thecompressed air stored within air tank 266 to pick inventory items for alimited time, and need only couple to supply system 138 when the robotdesires to refill the air tank. As a result, robot 200 can temporarilyoperate picking arm 206 on the grid without coupling to supply system138 and/or when the robot is driven off of the grid to assist with othergrasping and sorting tasks.

FIGS. 12A and 12B illustrate an example embodiment of picking arm 206coupled to pneumatic gripping tool 248. Picking arm 206 is moveable withseveral degrees of freedom to position pneumatic gripping tool 248relative to inventory stored in any location within a container 110 andhas long stroke (in the Z-direction) to allow robot 200 to lift anysized item from the container and to deposit the item in order bin 214.For example, picking arm 206 may include at least six degrees offreedom. In one non-limiting example, picking arm 206 includes 3Cartesian degrees of freedom, a fourth degree of freedom in the yaw andfifth and sixth degrees of freedom for pitch and roll or to increaselinear stoke in the z-direction. In the illustrative embodiment, pickingarm 206 may include a base member 226, one or more horizontal extensions228, a vertical extension 230 and a positioning arm 232 configured toremovably secure pneumatic gripping tool 248. Positioning arm 232 may bea relatively thin tube that has a smaller diameter than gripping tool248. This allows positioning arm 232 to freely position gripping tool248 within container 110 without interference from other items orpartitions disposed within the container. Positioning arm 232 may becoupled to vertical extension 230 via a coupling mechanism 233 thatallows the positioning arm to translate along a “first linear pathway,”such as a track, extending along the length of the vertical extension.One or more fluid lines 253 (FIG. 13B) are disposed within positioningarm 232 to fluidly couple gripping tool 248 and coupler 222 (FIG. 11).If more than one fluid line 253 is utilized, the fluid lines may beindependent from one another.

Base member 226 may be attached to the vehicle body 202 and extend abovethe open top end 212 of the vehicle body. Base member 226 may include a“second linear pathway” 227, such as a track, extending along the lengthof the base member. Horizontal extensions 228 may be coupled to basemember 226 in a manner that allows the horizontal members to move alongthe second linear pathway 227 to vertically position vertical extension230. Horizontal extensions 228 are also rotationally coupled to basemember 226, one another, and vertical extension 230 via joints 236,actuators and motors (not shown) that allow the pneumatic gripping toolto be positioned relative to the product items with several degrees offreedom. In an exemplary embodiment, the actuators may have magneticencoders with diametrically polarized magnets coupled to the motorrotor. The motor may be in the form of a brushless motor and have alarger diameter than length. Picking arm 206 may alternatively bepneumatically or hydraulically actuated and utilize actuatable valves tocontrol hydraulic or pneumatic rotary or linear actuators that controlthe pose of pneumatic gripping tool 248. As will be further explainedwith reference to FIG. 12C, in a preferred embodiment, the combinationof the first and second linear pathways is equal to or greater than 2times the height of containers 110, and preferably equal to or greaterthan 3 times the height of the containers.

FIG. 12C is a schematic, cross-section view illustrating a targetcontainer 110 b holding a first, relatively small item 238 (in height)and a second, relatively large item 240 (in height) that isapproximately the height of the container. The long stroke picking arm206 of robot 200 is capable of picking both item 238 and item 240 fromtarget container 110 b (when the target container is positioned at thetop layer of a stack 112 and just below grid 126) and depositing theitems in order bin 214. For example, in picking item 238 from the bottomof target container 110 b, gripping tool 248 must first be lowered adistance equal to the height of the robot body (shown in FIG. 9A) andthen lowered approximately the height of target container 110 b (e.g., adistance equal to approximately 2 times the height of the targetcontainer). Thus, in order to pick item 238, horizontal extensions 228may be moved to the bottom of first linear pathway 227 and positioningarm 232 may be moved downward relative to vertical extension 230 and tothe bottom of the second linear pathway to allow gripping tool 248 tocontact and grasp the relatively small item 238. After item 238 has beengrasped, positioning arm 232 may be moved upward and to the top of thefirst linear pathway and the horizontal extensions may be moved upwardsalong the second linear pathway 227, allowing the first item to bedeposited within order bin 214.

On the other hand, to grasp the relatively large second item 240 and todeposit the second item in order bin 214, the gripping tool must beraised to a sufficient height that allows the bottom of the second itemto clear the top of the order bin (e.g., gripping tool 248 must bepositioned a distance equal to approximately the height of the containerabove the top of the order bin). Thus, after the relatively large seconditem 240 has been grasped by picking arm 206, positioning arm 232 may beretracted upwards relative to vertical member 230 along the first linearpathway and the horizontal members may move toward the top of secondlinear pathway 227 to allow the bottom of the second item 240 to clearthe top of order bin 214. Thus, it will be appreciated that in order tograsp and deposit relatively small items such as item 238 and relativelytall items such as item 240, the stroke of picking arm 206 (in theZ-direction) must be at least 2 times the height of the containers andpreferably 3 times the height. While the stroke length may beaccomplished with a single linear pathway, dividing the stroke lengthinto two or more linear pathways allows picking arm 206 to be morecompact and have a smaller vertical profile.

It will be appreciated that further increasing the stroke of picking arm206 in the z-direction will allow the picking arm to reach underneathgrid 126 and pick items from a target container 110 b located on the topof a stack but beneath the upper most level in the same manner asdescribed above so long as no non-target containers 110 a lie on top ofthe target container.

In a preferred embodiment, as shown in FIG. 12B, picking arm 206 alsoincludes a spring 257 (and/or a back-drivable actuator, or a forcecontrolled actuator such as a quasi-direct-drive, direct-drive,series-elastic actuator, or geared actuator with torque sensing thatexhibits active compliance and functions as a virtual spring) or acompliant gripping tool 248 that provides passive or active complianceand that can be used to sense collisions and assist in performingmanipulation tasks such as picking or dense packing. The spring 257(and/or back-drivable actuator or force controlled actuator) may beprovided between pneumatic gripping tool 248 and positioning arm 232,and/or at the coupling mechanism 233 that couples positioning arm 232and vertical extension 230. Thus, if gripping tool 248 presses against aproduct or infrastructure of the storage structure with too great aforce, the gripping tool or the positioning arm 232 will recoil toprevent damage to the picking arm 206 and/or the product. The compliancemay also better position gripping tool 248 relative to the item duringgrasping.

It will be understood that picking arm 206 may be alternativelyconstructed and/or include fewer or additional components, so long asthe pneumatic gripping tool is positionable with several degrees offreedom to grasp inventory items stored within target container 110 b.For example, picking arm 206 may also include a load cell or aforce-torque sensor to measure the payload of a grasped item and/orsense an external force applied to the gripping tool. In this manner,robot 200 can instantaneously determine and/or verify the identity ofthe grasped item to pick and densely pack inventory items.

As mentioned above, gripping tool 248 is in fluid communication withcoupler 222 and thus in selective communication with fluid source S. Inembodiments in which fluid source S is a pneumatic compressor providingcompressed air, robot 200 may include one or more air ejectors, airaspirators, Venturi pumps 244 (FIG. 11) or similar devices (hereinafter“Venturi pump”) capable of using the compressed air of supply system 138to produce a vacuum or suction force. With reference to FIG. 13C anexample pneumatic circuit is shown, one or more “bypass valves” may beprovided between coupler 222 and gripping tool 248. The bypass valvesare controlled by one or more valve actuators and are transitionablebetween 3 conditions: a closed condition, a first open condition and asecond open condition. In the closed condition, the bypass valveprevents the compressed air from passing to gripping tool 248. In thefirst open condition, the bypass valve allows compressed air to flowfrom coupler 222, through Venturi pump 244 and to gripping tool 248.Thus, when the bypass valve is in the first open condition, the valveallows compressed air to flow through Venturi pump 244 such that theVenturi pump can generate a suction force to operate a gripping tool 248that relies on suction for grasping. In the second open condition, thebypass valve allows compressed air to flow from coupler 222 to grippingtool 248 but diverts the compressed air around Venturi pump 244. Thus,when the bypass valve is in the second open condition, the compressedair bypasses the Venturi pump and allows robot 200 to utilize thecompressed air to actuate a pneumatic gripping tool 248 such as clampand/or one of the other tool elements discussed below. The compressedair may also be utilized to blow or dispel air from gripping tool 248 toreposition inventory items within containers 110 and/or to repositioninventory items within order bins 214 to facilitate packing. Additionalvalves (“variable valves”) such as a throttle regulator may be providedupstream of the bypass valve (e.g., between coupler 222 and the “bypassvalve”) to precisely regulate airflow to the bypass valve and, in turn,to gripping tool 248. The variable valves and the bypass valves may besolenoid valves and may be selectively activated by a driver or relaycontrolled by a processor.

Referring to FIG. 13A, positioning arm 232 includes a magnet 246, forexample, a ring magnet 246 or a magnet arrangement that magneticallycouples gripping tool 248 to the positioning arm. Gripping tool 248 maybe any pneumatically actuated tool for grasping items. For example,gripping tool 248 may be a suction cup having a sidewall 251 formed of aresilient material such as rubber with bellows 250, and a groove 249positioned above the bellows. The sidewall 251 of gripping tool 248 isthus adapted to compress when the gripping tool engages an object.Gripping tool 248 may further include a lip 252 formed from a resilientmaterial, which also may be a rubber, such that the lip of the grippingtool is adapted to deform to and create a seal with the surface of aproduct in which it engages. A ring magnet 254 may be provided ongripping tool 248 to attract the magnet 246 of positioning arm 232 andto magnetically couple the gripping tool to the positioning arm. Agasket, such as an O-ring 256, may be provided on gripping tool 248 toseal the connection between positioning arm 232 and the gripping tool.In some embodiments, gripping tool 248 may additionally have a groove(not shown) to cooperate with a protrusion feature (not shown) on thepositioning arm 232 to prevent rotational and axial movement of thegripping tool relative to the positioning arm when the gripping tool iscoupled to the positioning arm. In other embodiments, gripping tool 248may be coupled to positioning arm 232 via a mechanical connection suchas a push/pull connection, snap fit connection, hook in slot connection,tab-in-slot connection, a twist/locked connection or any combination ofmale/female mechanical connections.

Turning now to FIG. 13B, gripping tool 248 may include one or moreelements 247 to assist in performing its gripping task. As used herein,the term “tool” means any device that is affixed to or coupleable to thepicking arm 206 of robot 200 and designed to perform a fulfillment tasksuch as grasping items, packing items, cutting boxes etc. In contrast,the term “element” denotes particular aspects of the tool as a whole.For example, gripping tool 248 may have one or more gripping elementssuch as a suction cup and/or fingers for grasping an item. In thisexample, the suction cup is an element and each of the fingers areindividual elements forming the entire tool. As shown in FIG. 13A, theentire tool 248 can be formed of a single element (e.g., the suctioncup). In other instances, it will be appreciated that tool 248 mayinclude multiple elements such as suction cups, fingers and/or otherelements for completing fulfillment tasks. The term “gripping tool” asused herein means that the tool includes at least one gripping elementprimarily designed to grasp items but does not exclude the tool fromhaving additional elements designed primarily to complete otherfulfilment tasks. Furthermore, the terms “pneumatic tool” and “pneumaticelement” mean that the tool or the element is pneumatically actuated.

An exemplary gripping tool 248 may include a suction cup and/or a clamp(not shown) having a plurality of pneumatically actuated fingers. Thefingers may be used in combination with the suction cup or in isolationof the suction cup to grasp products. In some embodiments, the fingersthemselves may include suction cups. In other embodiments, as shown inFIG. 13B, gripping tool 248 may include a plurality of suction cupsarranged on a single gripping tool. The plurality of suction cups may bearranged in an array to grip large and heavy inventory items at severaldiscrete locations, thereby providing a more stable grasp than a singlesuction cup. The suction cups may also be arranged to grasp multipleitems at once. In further embodiments, other gripping elements may beutilized. By way of example, these gripping elements may includeuniversal jamming grippers, foam vacuum grippers, pneumaticallyinflatable fingers, variable stiffness fingers, pressure actuatedfingers, pneumatically actuated linkage or piston driven grippers withrigid or compliant fingers or any other pneumatically driven or vacuumdriven (positive or negative pressure) gripper elements. Gripping tool248 can also include conductive target pads and push-pins on thegripping side of the gripping tool (or vice-versa) to provide power andcommunication signals to internal sensors and/or actuators of thegripping tool or to electrically supplement the pneumatic grasp. Inother embodiments, the tool need not include a gripping element forgrasping items. The tool may instead include an element such as a knife,a pneumatic rotary cutting tool (shown in FIG. 9G) or anotherpneumatically actuated cutting tool to cut open boxes. The tool mayinclude any pneumatically actuated elements including but not limited toair caulk guns, spray gun, air chisels and punches, air cut-off tools,air drills, air files, air grinders and sanders, air guns, air hammers,air nailers, air nibblers and shears, air riveters, air routers, airscarifiers, air screwdrivers, air staplers, air tapping tools,air-powered saws, and air-powered ratchets and wrenches.

As shown in FIG. 13B, the positioning arm 232 of picking arm 206 mayinclude a plurality of discrete fluid lines 253 that are individuallycoupleable to one or more elements 247 on a single tool 248. Forexample, when the picking arm 206 of robot 200 is coupled to a grippingtool having a single element such as a single suction cup, each of thefluid lines 253 may be in communication with the single suction cup. Onthe other hand, when the picking arm 206 of robot 200 is coupled to agripping tool having a plurality of elements such as a plurality ofsuction cups, each fluid line 253 may be in communication with arespective suction cup allowing the suction cups to be independentlyactuated. Each fluid line 253 may have a Venturi pump 244, bypass valveand one or more variable valves associated with the fluid line 253 tocontrol the suction force or the force of compressed air as explainedabove with reference to FIG. 13C. As is shown in FIGS. 13B and 13C,multiple fluid lines 253 may provide a pneumatic supply to each elementon a multi-element tool for actuation or gripping purposes. For example,a variable stiffness finger may be in communication with 2 fluid linesto independently control finger position and finger stiffness.Alternatively, multiple fluid lines 253 may be in fluid communicationwith a single element tool such as a tool with one suction cup toprovide a higher flow rate to that tool.

Any one of the pneumatic elements described above, or a combinationthereof, may be used to grasp one or more objects at a time, packgrasped objects, swap battery packs on the robot, activate bomb baydoors on a bin, lift and attach an order bin to a container, cut or sealboxes, manipulate items within an order bin, for example, by nudging,blowing or toppling the items, or perform any other tasks thatfacilitate order fulfillment.

Referring to FIGS. 9A, 9F and 9G, the vehicle body 202 of robot 200 mayinclude a tool holder 258 for holding a plurality of tools 248. Toolholder 258 may include a plurality of retainers 260 a, 260 b(collectively “retainers 260”) such as arcuate or rectangular cutoutsfor receiving the groove 249 of gripping tool 248, or a holding areasuch as a cup for receiving the bottom or sidewall 251 of the grippingtool. In this manner, a plurality of different tools 248 (e.g., toolshaving different tool elements and/or number of or configurations oftool elements, or suction cups having lips of different sizes,materials, shapes, configurations or orientations) may beinterchangeably coupled to positioning arm 232 and to tool holder 258when not in use, such that the picking arm 206 can select a particulargripping tool based upon the size, shape, material or weight of theproduct in which the robot is tasked with grasping. In some embodiments,tool holder 258 may be magnetic to assist in securing tools 248. Toolholder 258 may alternatively, or additionally, include a compliantmember to secure tools 248 within retainers 260 via a snap-fitconnection. Each of one of the tools 248 may include an RFID, AR tag,calibrated weight, QR code or similar identifier capable of beingidentified by a sensor or the load cells of picking arm 206. In thismanner, robot 200 can determine if tool 248 is secured to picking arm206 and can verify that the secured tool is the desired tool.

In other embodiments, tool holder 258 may be provided at dedicated “toolholder stations” on or adjacent to particular areas of grid 126 or atother areas within warehouse 101. Robot 200 may thus drive to a toolholder station to swap individual tools or to swap one tool holder 258for a completely different tool holder having a different set of tools.In this manner, robot 200 can swap tool holders based upon its upcomingset of tasks such that the robot does not need to carry each tool thatit may ever be instructed to utilize.

Referring back to FIG. 9B, one or more sensors 264, such as a scanner,may be positioned on the vehicle body 202 or the picking arm 206 ofrobot 200 to scan picked products and determine and/or verify whichorder bin 214 the picked product should be deposited. The scanning fieldof the scanners may be multiplied by positioning mirrors on the innersurfaces of sidewalls 208. Sensors 264 or another sensor may be used tocapture an image or data of a grasped product after it has been picked(and before it has been deposited in order bins 214) to identify and/ordetermine the size and dimensions of the item. This information can betransmitted to remote computer 103 which can then instruct manipulatorrobot 200 to deposit the grasped item within a particular location ofone of order bins 214 and/or in a particular orientation to facilitatedense packing. In some embodiments, vehicle body 202 may also include ashelf or ledge-like feature upon which a grasped item may be temporarilyplaced and, subsequently re-grasped from a different orientation, tofacilitate packing based upon the properties of the grasped item and theother items needing to be subsequently placed in that container.Alternatively, a gripping tool 248 having a plurality of grippingelements 247 with one or more gimbal degrees of freedom between theelements may be manipulated to adjust the gripping elements relative toone another to re-grasp the earlier grasped item in a desiredorientation. It will be appreciated that the grasping, re-grasping,packing, or any other manipulation task of one or more items (orcontainers) may also be accomplished using two or more gripping tools248 provided on two or more robots 200.

Use of robotic system 100 to piece pick individual product items fromcontainers 110 will now be described. Robot 200 may use its picking arm206 to grasp one or more order bins 214 and attach the bins to its ownvehicle body 202 or the vehicle body of another robot. Alternatively,order bins 214 may be attached to robot 200 by the digging plate oranother device on the robot, or external to the robot, or with theassistance of an operator. Robot 200 may then be autonomously positionedon grid 126 and operated under the control of a remote computer 103,which continuously logs the location of each of the robots, containers110 and products contained within the containers. Remote computer 103 isadditionally designed to efficiently control the movement of robots 200and may employ a series of safety checks to teleoperator instructionsand autonomous commands to prevent the robots and robotic systemsdescribed herein from colliding with one another as they move about thewarehouse.

When one or more orders are received, the computer assigns the orders toone or more of the manipulator robots 200 based upon the current ordervolumes of each of the robots and the locations of the productscontained in the order. If the product is located beneath one or morenon-target bins 110 a, robot 200, or a separate digging robot 205located nearby, may pull target bin 110 b to the top of stack 112. Forexample, digging robot 205 may position itself over a stack 112containing the target bin 110 b. Digging robot 205 may then extenddigger 207 underneath the digging robot and between vertical members 116and stack 112 (on a single or both lateral sides of the stack) to graspthe target bin 110 b and each of the non-target bins 110 a positionedbetween the target bin and grid 126. Each of the grasped bins may thenbe lifted such that the non-target bins are lifted, for example, throughthe receiving cavity of the digging robot and the target bin ispositioned within the receiving cavity. Digging robot 205 may then driveto a location over a separate stack 112 that is missing a singlecontainer and release target bin 110 b on the top of that stack suchthat manipulator robot 200 can pick items from the target bin. Inreleasing target bin 110 b, digging robot 205 may release only thetarget bin (e.g., never release non-target bins 110 a) or release thetarget bin and the non-target bins to stack the non-target bins on topof the target bin such that the bottom most non-target bin is positionedwithin the receiving cavity of the digging robot and the othernon-target containers are stacked above the receiving container of thedigging robot, before and again securing all of the non-target bins,driving back to the original stack and depositing the non-target bins inthe original stack and in the original order, less the target bin.

With target bin 110 b at the top of stack 112, the remote computer 103then autonomously directs the assigned robot 200 to a first position ongrid 126 located above or adjacent to the target bin. Mobility assembly204 allows robot 200 to navigate rails 122, 124, 125 and move to thedesired position on grid 126. Robot 200 may then transition valve 150 toits open condition to receive pneumatic air to pick from target bin 110b.

More specifically, as is shown in FIG. 14A, coupler 222 is positionedwithin cavity 143 such that the magnet 223 of the coupler engages themagnet 157 surrounding port 146. Insertion of coupler 222 into cavity143 may be aided by the tapered edges of the coupler and the taperededges of the cavity. In this manner, if coupler 222 is slightlymisaligned with respect to port 146, the tapered edge of the couplerwill slide down the tapered edge of the cavity to guide the coupler intoproper alignment with the port.

Alignment may also be aided by the magnetic connection between themagnet 223 of coupler 222 and the magnet 157 or ferrous materialsurrounding port 146. The magnetic connection also aids in securingcoupler 222 within cavity 143 and against the upwardly directed force ofcompressed air that is created as device 224 compresses plug 154 intoconduit 144 (away from the upper surface 128 of rails 122, 124, 125)while valve 150 is transitioned to the open configuration, therebyallowing pneumatic air to flow around the plug and into the coupler. Inthe event that the valve is an electrohydraulic servo valve, the couplermay be similarly engaged with the valve such that the conductive targetpads provide power to electrically transition the valve from the closedcondition to the open condition. The electrohydraulic valve mayalternatively be transitioned by a voltage received from grid 126 uponreceiving a signal from robot 200 or remote computer 103.

With fluid supply system 138 coupled to robot 200, the robot mayimmediately use the compressed air for grasping and/or store thecompressed air within air tank 266 for later use. In embodiments inwhich pneumatic gripping tool 248 relies on a suction force to graspobjects, the one or more Venturi pumps 244 can use the compressed airprovided by pneumatic air source S to generate a suction force foroperating gripping tool 248.

Upon arrival at a desired grid space 127, the picking arm 206 andpneumatic gripping tool 248 may immediately be positioned in thegrasping pose as instructed by the remote computer 103, as explainedabove with reference to FIG. 9C, or as instructed by a teleoperator.

The method of grasping a product item will now be explained with furtherreference to FIG. 15 and flowchart 500. If robot 200 has notpredetermined the grasping pose before the robot 200 is in the pickingposition, the method will begin, at 502, with a command from processor103 that instructs sensors 262 to capture an image of the inventorydisposed within a target container 110 b. After manipulator robot 200receives the selected grasping pose signal, the robot executes thesignal, at 504, causing picking arm 206 to perform the selected gaspingpose. That is, gripping tool 248 approaches the product item, asinstructed by processor 103, and contacts grasping region 414 of theproduct item. After the grasping attempt, one of the sensors such as apressure sensor (shown in FIG. 13C), characterizes the grasp, at 508, aseither successful or unsuccessful. That is, if the picking arm 206 ofrobot 200 is able to successfully grasp and remove the product item fromtarget container 110 b, the pressure sensor will characterize the graspas successful and transmit a successful grasp signal to processor 103via communication channels 104. On the other hand, if the picking arm206 of robot 200 is unable to remove the item from the container, or thepicking arm drops the item before the processor 103 instructs robot 200to release the item, the pressure sensor will characterize the grasp asunsuccessful and transmit an unsuccessful grasp signal to the processorvia communication channels 104. Upon characterizing the grasp asunsuccessful, processor 103 can either: (1) immediately signal toteleoperator interface 102, at 510 a, and request operator intervention;or (2) attempt to determine a new or modified grasping pose, at 510 b,to autonomously pick up the product item based upon the new or modifiedgrasping pose. If processor 103 elects to autonomously determine agrasping pose, the steps described above, with respect to FIG. 9C, maybe repeated until either the grasp is characterized as successful, at512, or until operator intervention is requested at 510 a.

FIG. 21 is a flowchart 2100 illustrating a high-level overview of anexample method for controlling the operation of a mobile piece pickingrobot, such as piece picking robot 200, by a computing system, such ascomputing system 103, and an operator interface, such as operatorinterface 102, when operator intervention is requested at 510 a. Asshown in block 2102, the computing system 103 receives performance datafrom a robot. The performance data may include inventory data relatingto an inventory item stored in a container within storage structure 114that the robot is tasked with manipulating or grasping. Performance datamay also include an assistance request from a robot indicating that therobot requires assistance in manipulating or otherwise grasping anobject or other performance statistics received from the robot, such asgrasp success rate, the number of consecutively failed grasps, etc. Thecomputing system 103 then transmits a notification to operator interface102, as shown in block 2104. The notification may be transmitted inresponse to receiving the performance data. The notification maycorrespond to the performance data and include inventory data or otherinformation regarding the inventory item the robot is tasked withmanipulating or grasping. The computing system receives controlinstructions, as shown in block 2106. The control instructions arereceived from the operator interface and may include at least one of apartial pose for a gripping element of the robot, an identifiedmanipulation or grasping region on the inventory item, or a selection ofa gripping element to be used by the robot for manipulating and/orgrasping the inventory item. The computing system then forwards thecontrol instructions to the robot for execution to manipulate or graspthe inventory item, as shown in block 2108.

FIG. 22 is a flowchart 2200 illustrating a high-level overview of anexample method for controlling the operation of a mobile piece pickingrobot, such as piece picking robot 200, by a system, such as operatorsystem 102. As shown in block 2202, the system outputs one or moreimages of an inventory item. The system receives control instructionsfor the robot based on the one or more images, as shown in block 2104.The system then forwards the control instructions to the manipulatorrobot, as shown in block 2106.

At a more detailed level, when processor 103 signals for intervention,the signal may be sent directly or indirectly to operator interface 102.In situations in which operator interface 102 is communicatively coupledto a plurality of manipulator robots 200, each of the robots may beindirectly coupled to operator interface 102 via a “broker”. The brokermay be part of processor 103, or a separate processor, tasked withordering the help requests from each robot within a queue of theoperator interface. The broker may run an algorithm to determine a“needs help score” to determine the priority of the queue or the brokermay connect a teleoperator directly to a particular robot based on the“needs help score” generated by the robot. The algorithm may be based onseveral factors including the number of prior grasp failures, elapsedtime from start of task, the level of task difficulty, the level ofprecision needed, the product/SKU to be manipulated, the task to beperformed (e.g., picking, packing, auditing inventory, or correctingother errors) and the like.

Once the help request signal has been received by operator interface102, an operator can remotely pilot the picking arm 206 of robot 200 anddirect the picking arm to execute a specified grasping pose to grasp theproduct item. Specifically, the operator can view the items on theoutput device (e.g., the display) of operator interface 102 and directlycontrol the picking arm 206 of robot 200 to grasp the grasping region414 of the item by manipulating the input device of the operatorinterface. In some instances, the operator may also prompt picking arm206 to grasp a product item in combination with an automated motionsequence calculated by a motion planner. In this manner, the operatormay simply select a pixel on the image feed representative of thegrasping region 414 while processor 103 autonomously determines andinstructs robot 200 to execute a selected grasping pose as describedabove with reference to FIG. 9C.

The pressure sensors or other sensors can then characterize the grasp aseither successful or unsuccessful as described above at 508. Theoperator can additionally, or alternatively, make the samecharacterization. If the sensor (or the operator) characterizes thegrasp as successful, the grasping data (e.g., grasping pose, graspingregion, gripping tool, inventory data, other sensor or robotinformation, etc.) used to grasp the product item may be saved withinstorage device 105, at 514, for future use. Robot 200 can thus learn toinfer or predict new grasping poses to improve automation of thegrasping process.

There is not a single gripping tool that can optimally handle a largevariety of inventory. For this reason, robot 200 may autonomouslydecide, or be instructed from the teleoperator, to switch grippingtools. Gripping tool 248 may be selected based upon the type of task orthe product type (which may be determined by the remote computer throughinventory tracking of the product types in each bin), analysis of theimage data and/or as a result of historical data relating to successfulpicks of that product or similar constructed products. Morespecifically, the remote computer 103, or an operator, may instructmanipulator robot 200 to couple a particular gripping tool 248 topicking arm 206 that can engage the grasping region 414 of the item withminimal leakage between the gripping tool and the surface of the item.

With reference to FIGS. 9F and 13A, if robot 200 or the teleoperatordetermines that it is desirable to switch gripping tools 248, the robotwill move picking arm 206 to position the groove 249 and/or the sidewall251 of the gripping tool attached to the picking arm within one of theretainers 260 of tool holder 258. Subsequently, the picking arm 206 maybe retracted or moved upward to decouple the magnet 246 of the pickingarm from the magnet 254 of gripping tool 248. The picking arm 206 maythen be positioned over another one of the gripping tools, positionedwithin tool holder 258, to magnetically couple the picking arm to theother one of the gripping tools before moving the picking arm laterallyto slide the coupled gripping tool out of its respective retainers 260.It will be appreciated, however, that other mechanical mechanisms forswapping gripping tools such as a push-pull connection or a twist-lockedconnection may also be utilized and the picking arm 206 of robot 200 maybe manipulated in any manner that facilitates disconnection of a firstgipping tool within the tool holder 258 and the connection of a secondgripping tool within the tool holder.

As gripping tool 248 is brought into contact with the product item, thelip 252 of the gripping tool deforms and conforms to the surface of theproduct as a suction force is applied to grasp the product.Additionally, the compliance in gripping tool 248 and/or picking arm 206will compensate for inaccuracies of the sensing system or graspingalgorithm to position the gripping tool in a better grasping pose uponcontact with the product. With the product grasped, picking arm 206 maythen lift the target product from container 110 and optionally positionor wave/rotate the product in front of scanners 264 to scan anidentifier such as a barcode or RFID located on the target product forthe purpose of confirming that the correct product has been graspedand/or to inform the picking arm as to which order bin 214 the productshould be released. During this time, one of the sensors mayadditionally collect data relating to the size and dimension of theproduct and transmit this information through communication channels 104to remote computer 103.

In some instances, remote computer 103 may then autonomously instruct,or the teleoperator may manually instruct, picking arm 206 to release orplace the grasped item in a particular location and/or orientationwithin order bin 214. Gripping tool 248 and/or other elements of thegripping tool may then be used to push, blow on, or otherwise manipulatethe product to a particular location or orientation within bin 214. Inthis manner, subsequently picked items may be efficiently packed withinorder bin 214 such that smaller order bins may be utilized. Thisincreases the overall amount of order bins that may be transported by asingle robot and, in turn, increases the throughput of the system. Whilethis disclosure has primarily described the processor (whether remote oronboard) as configured to implicitly or explicitly analyze images andexecute machine learning algorithms and policies for the purpose ofpredicting grasping poses, determining grasping regions or desiredgipping elements/tools, it will be appreciated that the processors areadditionally configured to implicitly or explicitly analyze images oforder containers 214 to determine packing poses, desired packing regionswithin the order bins, or desired packing tools that facilitate densepacking. Similar algorithms and analysis can be used to assist in theperformance of other manipulation tasks. Finally, these images and/orteleoperator commands in response to the images may be saved withinstorage device 105 as being associated with a particular manipulationtask for future use. Robot 200 can thus learn to infer or predict how toperform manipulation tasks (e.g., grasping or packing).

After robot 200 has sequentially picked up each of the productscorresponding to a particular order, order bins 214 may be transportedout of storage structure 114, for example, via shafts 121 and theassociated conveyor belts, for additional processing, sorting, packagingand/or shipping. If robot 200 is tasked with picking multiple consumersorders at once, robot 200 need not pick all of the products pertainingto the first consumers order before beginning to pick the secondconsumer's orders. In fact, the remote computer will direct robot 200 topick items based upon the storage locations of the products irrespectiveof the consumer who ordered the product and in an order that willfacilitate dense packing of the items.

FIG. 16A shows a mobile, manipulator robot 600 (sometimes referred tohereinafter as “manipulator robot” or “robot”) according to anotherembodiment of the present disclosure. Manipulator robot 600 may includeany of the features described above in connection with robot 200 and anyof the additional features described below. Common features betweenmanipulator robot 200 and manipulator robot 600 are not described againin detail hereinafter. Instead, when such features are discussed inconnection with manipulator robot 600, the features are merelyreferenced with a corresponding 600 series numeral. For example,manipulator robot 600 includes a mobility assembly 604 and a picking arm606 which may respectively be configured as described above with respectto mobility assembly 202 and a picking arm 206 in connection withmanipulator robot 200.

Manipulator robot 600 includes a vehicle body 602 which may be formed offour sidewalls 608. The vehicle body 602 of manipulator robot 600 mayhave an open or closed bottom end 610 and an open or closed top end 612.The sidewalls 608 are preferably sized such that vehicle body 602 has afootprint of a single grid space 127. In other words, when robot 600 ispositioned on the horizontal grid 126, two opposing sidewalls arepositioned over two adjacent rails 122 extending in the X-direction,while the other two opposing sidewalls are positioned over two adjacentrails 124 extending in the Y-direction. In other embodiments, thevehicle body 602 of robot 600 may have a footprint that is larger than asingle grid space 127. For example, the vehicle body 602 of robot 600may have a footprint equal to 1×2 grid spaces, 2×2 grid spaces 3×3 gridspaces. Hardware and other components may be stored within a cavity ofthe vehicle body. For example, the cavity of vehicle body 602 may housea small air tank and/or a relatively small battery 667 orsuper/ultra-capacitor and/or a heating element.

Picking arm 606 of robot 600 may be configured to engage and disengagebattery 667 to a conductive contact 669. In this regard, when battery667 is low, picking arm 606 can disconnect the battery from conductivecontact 669 and place the battery on a charging station (not shown).Picking arm 606 can then grab a charged battery from the chargingstation (not shown) and place the charged battery into contact withconductive contact 669 to transfer power from the charged battery orsuper/ultra-capacitor to the robots various drive mechanisms. Batterypacks may also be “swapped” or exchanged without using picking arm 606.For example, battery packs may be swapped by moving robot 200 in a firstdirection to bring the battery into a secured engagement with a “batteryswap port” (not shown) and manipulating the robot relative to thebattery swap port in a manner that disengages the battery from theconductive contact and allows the robot to drive away from the batteryswap port without the battery. In one example, after battery 667 hasbeen engaged with the battery swap port, the plunger of the propmechanism may be extended which, in turn, lifts the body of the robotand causes the battery to disengage its conductive contact 669.Alternatively, a battery swap mechanism onboard or external to the robotmay be used to swap battery packs.

One or more container retrieval devices 668 may be permanently affixedor detachably coupleable to vehicle body 602. In other words,manipulator robot 600 may autonomously add or remove container retrievaldevices as desired and may carry zero to four container retrievaldevices at any one time (as shown in FIG. 25). In this regard, whenmanipulator robot 600 includes multiple container retrieval devices, thecontainer retrieval devices can be used to simultaneously carry multipleorder bins, simultaneously perform multiple digging operations (e.g.,including replacing earlier extracted containers back into the stacks)and/or perform a combination of the foregoing.

Container retrieval device 668 includes a pair of opposing support arms670 affixed to or coupleable to vehicle body 602 and a hoist plate 672designed to engage and secure containers 110. With additional referenceto FIG. 16B, hoist plate 672 has an open lateral side and defines anaperture 674 extending through its upper and lower surfaces such thatpicking arm 606 can access the interior of a container secured by thehoist plate. Aperture 674 is preferably slightly larger than the outerperimeter of containers 110 to allow hoist plate 672 to slide about astack 112 of containers 110. In this regard, as hoist plate 672 islowered along a stack 112 of containers 110, the stack of containerswill automatically align the hoist plate relative to the containers in alateral direction.

Hoist plate 672 may be coupled to and suspended from support arms 670 bycables 676 which are connected to a winding mechanism 678 such as aspool, hoist, or winch of container retrieval device 668. The cables 676can be wound and unwound or spooled into or out from support arms 670 toadjust the hoist plate 672 with respect to the support arms in thez-direction. An encoder 680 may be coupled to the spool or windingmechanism to measure the distance hoist plate 672 moves in thez-direction. The spool or winding mechanism may also include a torquesensor 682 to measure the weight of a container 110 supported by hoistplate 672 or to detect when a container 110 is in contact with stack112. Alternatively, cables 676 or support arms 670 may include loadcells, force sensors, strain gauges, or other sensor configured todetect the weight of a container. As a result, inventory audits can beperformed autonomously while a container is being lifted or held byhoist plate 672 to determine or confirm the number of product items thathave been removed from the container or to ascertain when the containeris running low on particular product types and needs to be replenished.Similarly, the sensors may be used to ensure that the manipulator robot600 does not attempt to lift one or more containers with a total loadgreater than it can handle.

Hoist plate 672 is adapted to engage with the top and/or one or moresides of container 110 to grip the container. For example, hoist plate672 may include sliding or pivotable hooks 686 that are engageable withthe rim of containers 110 and/or engagement features 689 such asapertures (shown in FIG. 24) formed in the rim or sides of thecontainers. The hooks are driven into engagement with a container 110 bya suitable drive mechanism housed within plate 672, which may be poweredand controlled by signals carried through cables 676, through a separatecontrol cable (not shown), or wirelessly. In one example, cables 676 maybe made from conductive metal strips to transfer power and electricalsignals between the hoist plate 672 and the support arms 670.

Container retrieval device 668 may further include a sensor 688 such asa camera, depth imager, or similar device to align the hoist plate tothe top of container 110. The sensor can use markers such as AR tags orbarcodes on containers 110, or otherwise use features of the containeritself, to facilitate proper alignment. Sensor 688 is preferably locatedon the hoist plate 672 but may also be located on support arms 670. Inaddition to facilitating alignment, the camera can continuously captureimages of adjacent grid spaces and, in turn, inventory stored inside ofadjacent storage containers as manipulator robot 600 traverses grid 126.These images may then be transmitted via network 104 to remote processor103 to assist in inventory auditing or to predict grasping poses for oneof the manipulator robots before that robot reaches target container 110b to increase throughput of the system. Moreover, sensor 688 may be usedto continuously track the items within an order bin 214 supported by thehoist plate 672. In this manner, when inventory items pertaining tomultiple orders are contained within a single un-partitioned order bin214, the items can continuously be tracked so that the processor knowswhich item is associated with which order so that the items can be latersorted into individual orders without having to scan each of the productitems.

In embodiments in which the container retrieval device 668 is detachablycoupleable to the vehicle body 602 of manipulator robot 600, themanipulator robot can autonomously swap (upon receiving controlinstructions from processor 103 or operator interface 102) one containerretrieval device having a first hoist plate for another containerretrieval device having a differently configured hoist plate. Eachcontainer retrieval device may have its own set of motors, actuators,sensors, processors, circuits, batteries and power systems and can matewith the vehicle body 602 of robot 600 using an electromechanicalinterface configured to transmit mechanical loads and electricalcommunications. For example, container retrieval device 668 with hoistplate 672, shown in FIGS. 16A and 16B, may be swapped for containerretrieval device 690 with hoist plate 692, shown in FIG. 16C, orcontainer retrieval device 694 with hoist plate 696, shown in FIGS. 16Dand 16E. Hoist plate 692 includes an array of suction cups (instead ofthe hooks 686 of hoist plate 672), each of which may be formed similarto the suction cup illustrated and described with reference to FIG. 13C,to engage and lift boxes, cartons and the like.

Hoist plate 696 may be similar to hoist plate 668 and may furtherinclude one or more suctions cups attached to the plate by an extendableand retractable arm 698. The arm 698 may also be laterally moveableabout the hoist plate in the X and Y directions. It will be appreciatedthat hoist plate 696 may be lowered and arm 698 may be extended (shownin FIG. 16D) to freely position the suction cup with multiple degrees offreedom within a target container 110 b located at any depth withinstorage structure 114 to grasp and pick individual items from the targetcontainer so long as there is not a non-target container 110 apositioned on top of the target container. The arm 698 may be retractedas shown in FIG. 16E (e.g., upwards in the z-direction) so as to notinterfere with hoist plate 696 engaging a container 110.

Any of the above described hoist plates may be outfitted with additionalsensors (e.g., a temperature sensor, a thermal camera, humidity sensorand like) to monitor the storage conditions within the various sectionsof storage structure 114 to verify that the sections are being regulatedappropriately based upon the product types being stored in that section.

The vehicle body 602 of manipulator robot 600 may contain a payloadmanagement system designed to transfer the payload from one or more ofthe container retrieval devices to the chassis of the vehicle body. Thepayload management system may rely on one or more actuators which may benon-backdriveable or that utilizes mechanical brakes to allow theactuator to be unpowered while holding and transporting bins.Alternatively, a separate non-backdriveable container engagementmechanism on the body of the robot may be used to engage with acontainer while the container is held by the hoist mechanism.

Container retrieval device 668, or any of the other container retrievaldevices mentioned herein, may also be swapped for a container retrievaldevice 768 including a pair of opposing arms 770 designed to directlyengage and a secure a container 110 (shown in FIG. 16F). Of course,container retrieval device 768 may alternatively be permanently affixedto the manipulator robot. Each arm 770 may be extendable away from thevehicle body in a lateral direction and include a hook or latch 772 adistal end of the arm which may be rotatable or pivotable such that thehooks surround and engage target container 110 b. In this manner, arms770 are designed to extend and retract in a horizontal direction todirectly engage a container 110 and pull the target container off of ashelf located within the warehouse (e.g., on a lateral side of storagestructure 114 or any another shelf located within the warehouse) so longas the container is located laterally adjacent to the body of the robotas shown in FIG. 16F.

Use of manipulator robot 600 will now be described only with referenceto container retrieval device 668 as manipulator robot 600 is otherwiseoperated as previously described above with respect to robot 200. Toretrieve a target container 110 b from a stack 112, manipulator robot600 is moved about grid 126 to position container retrieval device 668over the stack containing the target container 110 b. The hoist plate672 may then be lowered by un-spooling cables 676 such that eachnon-target container 110 a (if any) passes through aperture 674 untilthe hoist plate is located adjacent to a non-target container locatedexactly one level above the target container 110 b. With hoist plate 672at the appropriate height, hooks 686 may then be slid towards thenon-target container located exactly one level above the targetcontainer 110 b to engage with the engagement features 689 or otherfeatures of that non-target container and to secure the container to thehoist plate.

Each of the non-target containers 110 a located on top of the targetcontainer 110 b may then be lifted by hoist plate 672 by spooling cablesupwards until the bottom most non-target container (the containersecured to the hoist plate) is located between support arms 670.Manipulator robot 600 may then be driven to a location over any otherstack and can release each of the non-target bins that it is carrying.Because hoist plate 672 is three sided (e.g., has an open side),manipulator robot 600 will be able to release all of the non-targetcontainers 110 a that the robot is carrying and move away the releasednon-target containers even if one or more of the containers is locatedabove grid 126. This would not be possible if hoist plate 672 was fullyenclosed. It will be appreciated that any of the hoist plates or diggingapparatuses described herein may have an open side similar to hoistplate 672. After manipulator robot 600 has released the stack ofnon-target container 110 a, the manipulator robot (or another robot) mayretrieve target container 110 b from the stack. Manipulator robot 600may pick items directly from the target container 110 b into an orderbin secured by another container retrieval devices 668 held by the robotor alternatively place the target bin 110 b on the top of another stackbefore it picks from the target bin and places the grasped item in anorder bin that the robot is carrying or that is otherwise nearby.

Alternatively, manipulator robot 600 can extract the target bin 110 band each of the non-target bins 110 a in a single lift. A single liftextraction may be accomplished by lowering hoist plate 672 around eachof the non-target bins, securing the latches 686 of the hoist plate tothe target bin and lifting the hoist plate until the target bin is heldbetween support arms 670 (e.g., within a container receiving cavity ofcontainer retrieval device 668) and the non-target bins are positionedabove the container retrieval device. Manipulator robot 600 may thenposition its container retrieval device 668 over another stack ofcontainers missing exactly one container before lowering and releasingall of the containers such that the target container 110 b is positionedjust beneath grid 126 (e.g., at the uppermost level where it can bepicked from) and each of the non-target containers are positioned on topof the target container and stacked above the grid. The hoist plate 672may then grab each of the overlying non-target containers 110 a and movethe non-target containers to any other stack, which again need not be astack missing containers in an amount equal to or greater than theamount of non-target containers secured by manipulator robot 600. Inother words, manipulator robot 600 can optionally release each of thenon-target containers simultaneously, even if one or more of thecontainers will be positioned above the grid, because the open side ofhoist plate 672 will allow manipulator robot 600 to drive away from thenon-target containers 110 a after they have been released to performother tasks. When manipulator robot 600 has more than one containerretrieval device 668, the container retrieval devices may independentlyor simultaneously perform a digging operation (as shown in FIG. 24).

In a variant aspect, a manipulator robot may include any and all of thefeatures of manipulator robot 200 and manipulator 600 but for theparticulars of its pneumatic system as discussed below. The pneumaticsystem 300 of the variant robot is schematically illustrated in FIG. 17.In this variant, the robot does not rely on pneumatic air from thestorage structure, instead the robot may have a modified pneumaticsystem coupled to the vehicle body of the robot. The pneumatic systemmay include a single or two-tiered vacuum having a first vacuum 302 anda second vacuum 304 in selective communication with a gripping tool suchas a single suction cup or a modified gripping tool 348 (FIG. 18). Firstvacuum 302 may be a vacuum with high flow rate (capable of displacinglarge volumes of air per minute) while second vacuum 304 may be a strongvacuum generator capable of producing a larger pressure differentialwith atmospheric pressure (which increases the payload or force that canbe held by the suction cup). Pneumatic system 300 further includes twovalves 306 a, 306 b (collectively “valves 306”), for example,servo-valves that may be toggled between an open condition and a closedcondition for controlling communication between the first and secondvacuums and gripping tool 248 or modified gripping tool 348.

As shown in FIG. 18, modified gripping tool 348 includes a first suctioncup 308 and a second suction cup 310, which may be concentricallypositioned within the first suction cup. The first suction cup 308 andthe second suction cup 310 are otherwise formed generally as previouslydescribed with respect to the suction cup of gripping tool 248, andtherefore, are not again described in detail. The only difference beingthat that modified gripping tool 348 is a dual suction cup as opposed tothe single suction cup of gripping tool 248. First vacuum 302, or thehigh flow rate vacuum, may be in selective communication with firstsuction cup 308, while second vacuum 304, or the high pressure vacuum,may be in selective communication with second suction cup 310. It isemphasized that the any of the battery swap mechanisms described withrespect to manipulator robot 200 and manipulator robot 600 may beincorporated in the variant robot with pneumatic system 300 because eventhe above-described single or two-tiered vacuum systems (or an onboardcompressor system) will utilize significantly more battery thanmanipulator robot 200 or manipulator robot 600 which access a pneumaticsupply from an external source.

Use of the pneumatic system 300 will now be described only withreference to the grasping task as the variant robot is otherwiseoperated as previously described above with respect to robot 200 and/orrobot 600. Before grasping a product, valve 306 a may be transitioned toits open position, providing the first suction cup 308 with a high flowrate vacuum suction force as the lip of the first suction cup deforms tocorrespondingly match the surface of the target product. After aninitial seal has been initiated, the high pressure vacuum line of vacuum304 is enabled by transitioning valve 306 b to the open condition. Thehigh pressure suction force enables the picking arm to support largerpayloads than would otherwise be possible with the high flow rate vacuumalone. In this manner, a firmer grasp may be provided. Of course, valves306 a and 306 b could both be set to their open positions during initialgrasping of the target product and until the robot desires to releasethe target product. Alternatively, valves 306 a and 306 b could betoggled back and forth and between open, closed and partially closedconditions in order to achieve a desired grasp of the target product.

By utilizing two relatively small vacuum sources, a high flow ratevacuum and a high pressure vacuum, to respectively create an initialseal and to firmly grasp products, the physical size of the vacuums maybe reduced such that the vehicle body of the robot need not be asdramatically modified. In this manner, the variant robot may be used topiece pick products stored within storage structure 114 or within framestructure 14 discussed with respect to the prior art.

FIG. 19 is a perspective view of a modified robotic system 100′configured to efficiently store a plurality of stacked containers 110′.Modified robotic system 100′ includes all of the above describedfeatures of system 100 and the additional features describedhereinafter. For example, additional rails 122′, 124′, 125′ may extendabove the grid 126 (supporting manipulator robot 200, manipulator robot600 and/or digging robot 205) and alone, or in combination withadditional support members, form a gantry frame that supports one ormore robotic picking arms 206′ equipped with a pneumatic gripping tools248′ in a manner that permits the picking arm 206′ to move about thegantry frame and piece pick inventory from containers 110′. In thismanner, compressed air may flow through rails 122′, 124′, 125′, and/orthe additional rails, to the pneumatic gripping tool 248′ of picking arm206′ for grasping products from containers 110′. Rails 122′, 124′, 125′,or the additional support members positioned above the grid, may alsoinclude one or more valves similar to valve 150, such that the valvesare accessible to manipulator robot 200 or manipulator robot 600(positioned on the grid) to allow manipulator robot or manipulator robot600 to selectively couple to the pneumatic supply system.

Alternatively, robotic picking arms 206′ may be fixed on a frame abovethe grid and digging robot 205 or another bin carrying robot maytransport a target container 110′ to the fixed picking arm, which maygrasp the desired item(s) and place the grasped items into an order bincarried by a transporter robot. In this manner, containers 110′ need notbe transported down the ports and back-and-forth from thepicking/sorting stations.

FIG. 20 is a perspective view of yet another alternative robotic system100″ configured to efficiently store a plurality of stacked containers.Robotic system 100″ includes all of the above described features ofrobotic system 100 and the additional features described hereinafter.Manipulator robot 200, or another manipulator robot, such as manipulatorrobot 600, may be positioned at a station on grid 126 (e.g., so as tonot move), configured to move only within a specific area of the grid orotherwise stationed or configured to move about the warehouse. Theserobots may be permanently or selectively coupled to supply lines 140″that hang from a structure, such as the ceiling of warehouse 101, orotherwise extend toward the surface of grid 126 or toward the floor ofthe warehouse to provide the robots with access to a pneumatic supplywhen the robot is located on the grid or otherwise positioned off thegrid within the warehouse, for example, on the warehouse floor. In someembodiments, supply lines 140″ may be retracted, for example, via a dragchain cable carrier, a cable reel retractor or similar device to managecable slack in the supply lines. Supply lines 140″ may additionallyinclude a power cord, or other mechanism, to supply a voltage to themanipulator robot when the robot is coupled to the supply lines. Acontainer carrying robot, such as digging robot 205 or robot 600, cantransport a container 110 to the manipulator robots stationed within aparticular area of the grid, before inventory is picked from thecontainer and deposited into other containers such as order bins 214 asdescribed above with respect robot system 100.

Referring to FIG. 23, it will be appreciated that pneumatic air can alsobe supplied to a manipulator robot, such as robot 200 or robot 600, whenthe manipulator robots are positioned off of grid 126 and in other areasof the warehouse to facilitate product manipulation and/or piece pickingand/or any other tasks such as packaging, unpackaging, fabrication, ormanufacturing tasks. For example, a pneumatic air supply line may beprovided within a shelf and/or within a warehouse floor and/or within aline or hose extending from a structure above the robot and downwardstowards the robot so long as it is accessible to the coupler of any oneof the manipulator robots disclosed herein. In this manner, when therobot is driven off of grid 126 to assist with other fulfillment taskssuch as picking inventory from shelving and/or packaging thepicked/sorted inventory at a picking/sorting station, the robot hasaccess to a pneumatic supply to actuate its pneumatic tools. As aresult, the same compact robots can be used in various areas of thewarehouse to accomplish various order fulfillment tasks and can bedriven in any direction across the warehouse floor without needing tocarry large onboard air compressors or being permanently tethered to aflexible supply lines that could tangle up with other supply linesduring movement of the respective robots.

FIG. 26 is flow chart 2600 illustrating the steps of an example orderfulfillment process. As shown in block 2602 inventory arrives at awarehouse, such as warehouse 101, via a truck which may drive up to thewarehouse dock door. The containers are unloaded from the truck, atblock 2604, either manually or using a robotic system within the truckand placed onto a conveyor. At block 2606, the conveyor or anotheroperator moves the container to an area within warehouse 101, where anyone of the robot described herein can use their container retrievaldevices to pick up the containers and transport them to a different anddesired area of the warehouse as represented by block 2608. To ship anitem from the warehouse, the inverse order is performed. In other words,the containers are transported by the container retrieval device of therobot, to the conveyor and then to the truck.

FIG. 27 is a flow chart 2700 illustrating the steps of another exampleorder fulfillment process. As shown in block 2702, a truck drivesinventory to a warehouse, such as warehouse 101, and arrives underneatha framed or “grid” structure similar to grid 126 described above. Thetruck may have a convertible top, a floor that is moveable relative toits top, or a removable pallet, pod or shipping container. At block2704, the truck may remove its top, move its floor relative to its top,or otherwise exposes its pallet, pod or shipping container. Any of therobots having container retrieval devices described herein may thendrive on the grid above the exposed containers and directly retrieve thecontainers from the truck bed, pallet, pod or shipping container, atblock 2706. Once again, to ship an item from the warehouse, the inversesteps are performed. More specifically, the robots may carry containersfrom the warehouse to the grid above the truck and lower the containersdirectly into the truck bed.

Although the disclosure herein has been described with reference toparticular embodiments, it is to be understood that these embodimentsare merely illustrative of the principles and applications of thepresent disclosure. It is therefore to be understood that numerousmodifications may be made to the illustrative embodiments and that otherarrangements may be devised without departing from the spirit and scopeof the present disclosure as defined by the appended claims.

1. An order fulfillment system, comprising: a mobile manipulator robotmoveable within a warehouse for picking inventory items, the mobilemanipulator robot comprising: a body; a wheel assembly coupled to thebody, the wheel assembly including a plurality of wheels and an actuatorto move the body within the warehouse; a sensor to locate a position ofthe body relative to the warehouse in which the body is located; aninterface configured to wirelessly send processor readable data to aremote processor and wirelessly receive processor executableinstructions from the remote processor; an imaging device to captureimages of the inventory items, the imaging device being configured tocollect inventory data; a picking manipulator coupled to the body, thepicking manipulator having at least three degrees of freedom; a firstpneumatic gripping tool formed of a compliant material and coupleable tothe picking manipulator, the first pneumatic gripping tool beingmoveable by the picking manipulator in a three dimensional workspace toaccess and grasp inventory items; and a container retrieval devicecoupleable to the body and having a container receiving space, thecontainer retrieval device including a hoist plate having a top surface,a bottom surface and an aperture extending through the top and bottomsurfaces, the hoist plate being extendable in a vertical directionrelative to the body to engage and move a container within the containerreceiving space.
 2. The order fulfillment system of claim 1, wherein thecontainer retrieval device includes a set of latches or hooks to engagethe container, the latches or hooks being slidable or pivotable relativeto the hoist plate.
 3. The order fulfillment system of claim 1, whereinthe hoist plate has an open side.
 4. The order fulfillment system ofclaim 1, further comprising a second pneumatic gripping tool coupleableto the picking manipulator, the second pneumatic gripping tool beingmoveable by the picking manipulator to access and grasp inventory items.5. The order fulfillment system of claim 4, wherein the mobilemanipulator robot further includes a tool holder with first and secondretainers for holding a respective one of the first and second pneumaticgripping tools.
 6. The order fulfillment system of claim 4, wherein themobile manipulator robot further comprises an onboard processor incommunication with at least one of the imaging device or the pickingmanipulator, wherein the picking manipulator is configured tointerchangeably couple to the first pneumatic gripping tool and thesecond pneumatic gripping tool upon receiving the processor executableinstructions from the remote processor or upon executing instructionsfrom the onboard processor.
 7. The order fulfillment system of claim 1,further comprising a storage structure and a plurality containers forstoring inventory items, the plurality of containers being arranged invertical stacks within the storage structure.
 8. The order fulfillmentsystem of claim 7, further comprising a tool holder coupled to thestorage structure, the tool holder having a plurality of retainers. 9.The order fulfillment system of claim 7, further comprising: a griddisposed above the containers, the grid including a first set ofparallel rails extending in a first direction and a second set ofparallel rails extending in a second direction substantiallyperpendicular to the first direction such that the first and second setof parallel rails collectively define grid spaces, the first and secondset of parallel rails having a profiled track to guide movement of thewheel assembly along the first set of parallel rails and the second setof parallel rails, wherein each vertical stack is arranged underneath arespective grid space, wherein the retrieval device is extendablebeneath the grid and within a respective grid space to engage and liftone of the plurality of containers within the receiving cavity.
 10. Theorder fulfillment system of claim 9, wherein the one of the plurality ofcontainers is a target container located underneath a plurality ofnon-target containers, and the aperture of the hoist plate is sized toallow the hoist plate to slide around a perimeter of each of thenon-target containers.
 11. The order fulfillment system of claim 9,wherein the container retrieval device is a first container retrievaldevice and is arranged on a first side of the body to lift a firstcontainer from a first stack, and wherein the mobile manipulator devicefurther comprises a second container retrieval device arranged on asecond side of the body to lift a second container from a second stackdifferent from the first stack.
 12. An order fulfillment system,comprising: a warehouse including a storage structure comprising: a gridincluding a first set of parallel rails extending in a first directionand a second set of parallel rails extending in a second directionsubstantially perpendicular to the first direction such that the firstand second set of parallel rails collectively define grid spaces; and astack of vertically arranged containers, the stack configured to bearranged underneath a respective one of the grid spaces; and a mobilemanipulator robot for picking inventory items stored within one of thecontainers, the mobile manipulator robot comprising: a body coupled to awheel assembly, the wheel assembly including a plurality of wheels andan actuator to move the body along the grid; a sensor to locate aposition of the body relative to the grid on which the body is located;an interface configured to wirelessly send processor readable data to aremote processor and wirelessly receive processor executableinstructions from the remote processor; an imaging device to captureimages of the inventory items, the imaging device being configured tocollect inventory data; a picking manipulator coupled to the body, thepicking manipulator having at least three degrees of freedom; first andsecond pneumatic gripping elements coupleable to the pickingmanipulator, each of the first and second pneumatic gripping elementsbeing moveable by the picking manipulator in a three dimensionalworkspace to access and grasp the inventory items stored within one ofthe containers, the first pneumatic gripping element comprising a firstsuction cup; a conductive contact to receive electrical power from anenergy source; and an onboard processor in communication with at leastone of the imaging device or the picking manipulator, wherein thepicking manipulator is configured to engage one of the first pneumaticgripping element or the second pneumatic gripping element with aninventory item after receiving the processor executable instructionsfrom the remote processor or executing instructions from the onboardprocessor.
 13. The order fulfillment system of claim 12, wherein themanipulator robot further comprises an onboard compressor or vacuum foroperating the first suction cup.
 14. The order fulfillment system ofclaim 12, wherein the energy source is a charged surface of the grid.15. The order fulfillment system of claim 12, wherein the energy sourceis an onboard battery.
 16. The order fulfillment system of claim 12,further comprising: a pneumatic supply line coupled to the storagestructure; and a plurality of valves in fluid communication with thepneumatic supply line, each of the valves being transitionable between aclosed condition and an open condition, and wherein the mobilemanipulator robot further comprises a coupler having a mating end influid communication with the suction cup, the mating end of the couplerbeing configured to selectively mate with one of the valves to access apneumatic supply from the pneumatic supply line.
 17. The orderfulfillment system of claim 12, wherein the second pneumatic grippingelement comprises a second suction cup, and wherein the mobilemanipulator robot further comprises a first fluid line extending betweenthe coupler and the first suction cup and a second fluid line extendingbetween the coupler and the second suction cup, wherein pneumaticpressure or flowrate within of the first fluid line is independentlycontrollable relative to pneumatic pressure or flowrate within thesecond fluid line.
 18. The order fulfillment system of claim 17, whereinthe first suction cup and the second suction cup are located on a singlepneumatic gripping tool.
 19. The order fulfillment system of claim 17,wherein the first suction cup is located on a first pneumatic grippingtool and the second pneumatic gripping element is located on a secondpneumatic gripping tool different than the first pneumatic grippingtool.
 20. The order fulfilment system of claim 17, further comprising afirst Venturi pump in fluid communication with the first fluid line anda second Venturi pump in fluid communication with the second fluid line.